SparkReceiverIO provides I/O transforms to read messages from an Apache Spark Receiver org.apache.spark.streaming.receiver.Receiver
as an unbounded source.
SparkReceiverIO supports Spark Receivers (Spark version 2.4).
RecordId
field for Salesforce and vid
field for Hubspot Receivers. For more details please see GetOffsetUtils class from CDAP plugins examples.To add SparkReceiverIO support for a new Spark Receiver
, perform the following steps:
build.gradle
file. Example: implementation "io.cdap:hubspot-plugins:1.0.0"
.Long offset
from the record of the Spark Receiver. Example: see GetOffsetUtils class from CDAP plugins examples.ReceiverBuilder
object by passing class of record that you want to read (e.g. String) and your Spark Receiver
class name (dependency from step 2). Example:ReceiverBuilder<String, HubspotReceiver> receiverBuilder = new ReceiverBuilder<>(HubspotReceiver.class).withConstructorArgs();
Receiver
with SparkReceiverIO:getOffsetFn
(from step 3) and correct ReceiverBuilder
(from step 4). Example:SparkReceiverIO.Read<V> reader = SparkReceiverIO.<V>read() .withGetOffsetFn(getOffsetFn) .withSparkReceiverBuilder(receiverBuilder);
To learn more, please check out CDAP Streaming plugins complete examples where Spark Receivers are used.
To use SparkReceiverIO, add a dependency on beam-sdks-java-io-sparkreceiver
.
<dependency> <groupId>org.apache.beam</groupId> <artifactId>beam-sdks-java-io-sparkreceiver</artifactId> <version>...</version> </dependency>
The documentation and usage examples are maintained in JavaDoc for SparkReceiverIO class.