| --- | |
| layout: global | |
| title: Extending Spark with Spark Server Libraries | |
| redirect_from: app-dev-spark-connect.html | |
| license: | | |
| Licensed to the Apache Software Foundation (ASF) under one or more | |
| contributor license agreements. See the NOTICE file distributed with | |
| this work for additional information regarding copyright ownership. | |
| The ASF licenses this file to You under the Apache License, Version 2.0 | |
| (the "License"); you may not use this file except in compliance with | |
| the License. You may obtain a copy of the License at | |
| http://www.apache.org/licenses/LICENSE-2.0 | |
| Unless required by applicable law or agreed to in writing, software | |
| distributed under the License is distributed on an "AS IS" BASIS, | |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | |
| See the License for the specific language governing permissions and | |
| limitations under the License. | |
| --- | |
| With its decoupled client-server architecture, Spark Connect simplifies how Spark Applications are | |
| developed. | |
| The notion of Spark Client Applications and Spark Server Libraries are introduced as follows: | |
| * _Spark Client Applications_ are regular Spark applications that use Spark and its rich ecosystem for | |
| distributed data processing. Examples include ETL pipelines, data preparation, and model training | |
| and inference. | |
| * _Spark Server Libraries_ build on, extend, and complement Spark's functionality, e.g. | |
| [MLlib](ml-guide.html) (distributed ML libraries that use Spark's powerful distributed processing). Spark Connect | |
| can be extended to expose client-side interfaces for Spark Server Libraries. | |
| With Spark 3.4 and Spark Connect, the development of Spark Client Applications is simplified, and | |
| clear extension points and guidelines are provided on how to build Spark Server Libraries, making | |
| it easy for both types of applications to evolve alongside Spark. As illustrated in Fig.1, Spark | |
| Client applications connect to Spark using the Spark Connect API, which is essentially the | |
| DataFrame API and fully declarative. | |
| <figure style="text-align: center;"> | |
| <img | |
| src="img/extending-spark-connect-labelled.png" | |
| alt="Extending Spark Connect Diagram" | |
| /> | |
| <figcaption>Figure 1: Architecture</figcaption> | |
| </figure> | |
| Spark Server Libraries extend Spark. They typically provide additional server-side logic integrated | |
| with Spark, which is exposed to client applications as part of the Spark Connect API, using Spark | |
| Connect extension points. For example, the _Spark Server Library_ consists of custom | |
| server-side logic (the blue box labeled _Plugin_, with its _App Logic_), which is exposed to | |
| the client via the blue _Proto API_ box as part of the Spark Connect API. The client uses this | |
| API, e.g., alongside PySpark or the Spark Scala client, making it easy for Spark client | |
| applications to work with the custom logic/library. | |
| Until Spark 3.4, extensions to Spark (e.g., [Spark ML](ml-guide#:~:text=What%20is%20%E2%80%9CSpark%20ML%E2%80%9D%3F,to%20emphasize%20the%20pipeline%20concept.) | |
| or [Spark-NLP](https://github.com/JohnSnowLabs/spark-nlp)) were built and deployed like Spark | |
| Client Applications. With Spark 3.4 and Spark Connect, explicit extension points are offered to | |
| extend Spark via Spark Server Libraries. These extension points provide functionality that can be | |
| exposed to a client, which differs from existing extension points in Spark such as | |
| [SparkSession extensions](api/java/org/apache/spark/sql/SparkSessionExtensions.html) or | |
| [Spark Plugins](api/java/org/apache/spark/api/plugin/SparkPlugin.html). | |
| * This will become a table of contents (this text will be scraped). | |
| {:toc} | |
| A Spark Server Library consists of the following components, numbered in Fig. 1: | |
| 1. The Spark Connect protocol extension (blue box _Proto_ API) | |
| 2. A Spark Connect Plugin. | |
| 3. The application logic that extends Spark. | |
| 4. The client package that exposes the Spark Server Library application logic to the Spark Client | |
| Application, alongside PySpark or the Scala Spark Client. | |
| ## Spark Connect Protocol Extension | |
| To extend Spark with a new Spark Server Library, developers can extend the three main operation | |
| types in the Spark Connect protocol: _Relation_, _Expression_, and _Command_. | |
| ```protobuf | |
| message Relation { | |
| oneof rel_type { | |
| Read read = 1; | |
| // ... | |
| google.protobuf.Any extension = 998; | |
| } | |
| } | |
| message Expression { | |
| oneof expr_type { | |
| Literal literal = 1; | |
| // ... | |
| google.protobuf.Any extension = 999; | |
| } | |
| } | |
| message Command { | |
| oneof command_type { | |
| WriteCommand write_command = 1; | |
| // ... | |
| google.protobuf.Any extension = 999; | |
| } | |
| } | |
| ``` | |
| Their extension fields allow serializing arbitrary protobuf messages as part of the Spark Connect | |
| protocol. These messages represent the parameters or state of the extension implementation. | |
| To build a custom expression type, the developer first defines the custom protobuf definition | |
| of the expression. | |
| ```protobuf | |
| message ExamplePluginExpression { | |
| Expression child = 1; | |
| string custom_field = 2; | |
| } | |
| ``` | |
| ## Spark Connect Plugin implementation with custom application logic | |
| As a next step, the developer implements the `ExpressionPlugin` class of Spark Connect with custom | |
| application logic based on the input parameters of the protobuf message. | |
| ```scala | |
| class ExampleExpressionPlugin extends ExpressionPlugin { | |
| override def transform( | |
| relation: protobuf.Any, | |
| planner: SparkConnectPlanner): Option[Expression] = { | |
| // Check if the serialized value of protobuf.Any matches the type | |
| // of our example expression. | |
| if (!relation.is(classOf[proto.ExamplePluginExpression])) { | |
| return None | |
| } | |
| val exp = relation.unpack(classOf[proto.ExamplePluginExpression]) | |
| Some(Alias(planner.transformExpression( | |
| exp.getChild), exp.getCustomField)(explicitMetadata = None)) | |
| } | |
| } | |
| ``` | |
| Once the application logic is developed, the code must be packaged as a jar and Spark must be | |
| configured to pick up the additional logic. The relevant Spark configuration options are: | |
| * `spark.jars` which define the location of the Jar file containing the application logic built for | |
| the custom expression. | |
| * `spark.connect.extensions.expression.classes` specifying the full class name | |
| of each expression extension loaded by Spark. Based on these configuration options, Spark will | |
| load the values at startup and make them available for processing. | |
| ## Spark Server Library Client Package | |
| Once the server component is deployed, any client can use it with the right protobuf messages. | |
| In the example above, the following message payload sent to the Spark Connect endpoint would be | |
| enough to trigger the extension mechanism. | |
| ```json | |
| { | |
| "project": { | |
| "input": { | |
| "sql": { | |
| "query": "select * from samples.nyctaxi.trips" | |
| } | |
| }, | |
| "expressions": [ | |
| { | |
| "extension": { | |
| "typeUrl": "type.googleapis.com/spark.connect.ExamplePluginExpression", | |
| "value": "\n\006\022\004\n\002id\022\006testval" | |
| } | |
| } | |
| ] | |
| } | |
| } | |
| ``` | |
| To make the example available in Python, the application developer provides a Python library that | |
| wraps the new expression and embeds it into PySpark. The easiest way to provide a function for any | |
| expression is to take a PySpark column instance as an argument and return a new Column instance | |
| with the expression applied. | |
| ```python | |
| from pyspark.sql.connect.column import Expression | |
| import pyspark.sql.connect.proto as proto | |
| from myxample.proto import ExamplePluginExpression | |
| # Internal class that satisfies the interface by the Python client | |
| # of Spark Connect to generate the protobuf representation from | |
| # an instance of the expression. | |
| class ExampleExpression(Expression): | |
| def to_plan(self, session) -> proto.Expression: | |
| fun = proto.Expression() | |
| plugin = ExamplePluginExpression() | |
| plugin.child.literal.long = 10 | |
| plugin.custom_field = "example" | |
| fun.extension.Pack(plugin) | |
| return fun | |
| # Defining the function to be used from the consumers. | |
| def example_expression(col: Column) -> Column: | |
| return Column(ExampleExpression()) | |
| # Using the expression in the Spark Connect client code. | |
| df = spark.read.table("samples.nyctaxi.trips") | |
| df.select(example_expression(df["fare_amount"])).collect() | |
| ``` |