PySpark <-> Spark message interface

This module implements message-based communication between PySpark and the Spark. It introduces abstraction layers, which handle the receiving and sending of messages. Through these abstractions, the underlying data transport channel (Unix domain socket, gRPC, etc.) can be decoupled from the core PySpark logic, which processes the data.

Overall, introducing these abstractions allows the same PySpark code to work with different underlying data transport channels transparently.

This module defines the following message types:

Spark -> PySpark

1. Initialization - UDF Payload, parameters, ...
2. Data - Data to invoke the UDF on
3. Finish - UDF has been invoked on all available data

PySpark -> Spark

1. Response data
2. Exceptions
3. Finish - All processing is done