docs/components/modules/ROOT/pages/flink-component.adoc

[[flink-component]]
= Apache Flink Component
:page-source: components/camel-flink/src/main/docs/flink-component.adoc

*Available as of Camel version 2.18*

This documentation page covers the https://flink.apache.org[Apache Flink]
component for the Apache Camel. The *camel-flink* component provides a
bridge between Camel connectors and Flink tasks. +
This Camel Flink connector provides a way to route message from various
transports, dynamically choosing a flink task to execute, use incoming
message as input data for the task and finally deliver the results back to
the Camel pipeline.

Maven users will need to add the following dependency to
their `pom.xml` for this component:

[source,xml]
------------------------------------------------------------
<dependency>
    <groupId>org.apache.camel</groupId>
    <artifactId>camel-flink</artifactId>
    <version>x.x.x</version>
    <!-- use the same version as your Camel core version -->
</dependency>
------------------------------------------------------------

== URI Format

Currently, the Flink Component supports only Producers. One can create DataSet, DataStream jobs.

[source,java]
-------------------------------------------------
flink:dataset?dataset=#myDataSet&dataSetCallback=#dataSetCallback
flink:datastream?datastream=#myDataStream&dataStreamCallback=#dataStreamCallback
-------------------------------------------------

[[Flink-FlinkEndpointOptions]]
FlinkEndpoint Options



// endpoint options: START
The Apache Flink endpoint is configured using URI syntax:

----
flink:endpointType
----

with the following path and query parameters:

=== Path Parameters (1 parameters):


[width="100%",cols="2,5,^1,2",options="header"]
|===
| Name | Description | Default | Type
| *endpointType* | *Required* Type of the endpoint (dataset, datastream). |  | EndpointType
|===


=== Query Parameters (8 parameters):


[width="100%",cols="2,5,^1,2",options="header"]
|===
| Name | Description | Default | Type
| *collect* (producer) | Indicates if results should be collected or counted. | true | boolean
| *dataSet* (producer) | DataSet to compute against. |  | DataSet
| *dataSetCallback* (producer) | Function performing action against a DataSet. |  | DataSetCallback
| *dataStream* (producer) | DataStream to compute against. |  | DataStream
| *dataStreamCallback* (producer) | Function performing action against a DataStream. |  | DataStreamCallback
| *lazyStartProducer* (producer) | Whether the producer should be started lazy (on the first message). By starting lazy you can use this to allow CamelContext and routes to startup in situations where a producer may otherwise fail during starting and cause the route to fail being started. By deferring this startup to be lazy then the startup failure can be handled during routing messages via Camel's routing error handlers. Beware that when the first message is processed then creating and starting the producer may take a little time and prolong the total processing time of the processing. | false | boolean
| *basicPropertyBinding* (advanced) | Whether the endpoint should use basic property binding (Camel 2.x) or the newer property binding with additional capabilities | false | boolean
| *synchronous* (advanced) | Sets whether synchronous processing should be strictly used, or Camel is allowed to use asynchronous processing (if supported). | false | boolean
|===
// endpoint options: END
// spring-boot-auto-configure options: START
== Spring Boot Auto-Configuration

When using Spring Boot make sure to use the following Maven dependency to have support for auto configuration:

[source,xml]
----
<dependency>
  <groupId>org.apache.camel</groupId>
  <artifactId>camel-flink-starter</artifactId>
  <version>x.x.x</version>
  <!-- use the same version as your Camel core version -->
</dependency>
----


The component supports 6 options, which are listed below.



[width="100%",cols="2,5,^1,2",options="header"]
|===
| Name | Description | Default | Type
| *camel.component.flink.basic-property-binding* | Whether the component should use basic property binding (Camel 2.x) or the newer property binding with additional capabilities | false | Boolean
| *camel.component.flink.data-set* | DataSet to compute against. The option is a org.apache.flink.api.java.DataSet type. |  | String
| *camel.component.flink.data-set-callback* | Function performing action against a DataSet. The option is a org.apache.camel.component.flink.DataSetCallback type. |  | String
| *camel.component.flink.data-stream* | DataStream to compute against. The option is a org.apache.flink.streaming.api.datastream.DataStream type. |  | String
| *camel.component.flink.data-stream-callback* | Function performing action against a DataStream. The option is a org.apache.camel.component.flink.DataStreamCallback type. |  | String
| *camel.component.flink.enabled* | Enable flink component | true | Boolean
|===
// spring-boot-auto-configure options: END




== FlinkComponent Options




// component options: START
The Apache Flink component supports 5 options, which are listed below.



[width="100%",cols="2,5,^1,2",options="header"]
|===
| Name | Description | Default | Type
| *dataSet* (producer) | DataSet to compute against. |  | DataSet
| *dataStream* (producer) | DataStream to compute against. |  | DataStream
| *dataSetCallback* (producer) | Function performing action against a DataSet. |  | DataSetCallback
| *dataStreamCallback* (producer) | Function performing action against a DataStream. |  | DataStreamCallback
| *basicPropertyBinding* (advanced) | Whether the component should use basic property binding (Camel 2.x) or the newer property binding with additional capabilities | false | boolean
|===
// component options: END





== Flink DataSet Callback

[source,java]
-----------------------------------
@Bean
public DataSetCallback<Long> dataSetCallback() {
    return new DataSetCallback<Long>() {
        public Long onDataSet(DataSet dataSet, Object... objects) {
            try {
                 dataSet.print();
                 return new Long(0);
            } catch (Exception e) {
                 return new Long(-1);
            }
        }
    };
}
-----------------------------------

== Flink DataStream Callback

[source,java]
---------------------------
@Bean
public VoidDataStreamCallback dataStreamCallback() {
    return new VoidDataStreamCallback() {
        @Override
        public void doOnDataStream(DataStream dataStream, Object... objects) throws Exception {
            dataStream.flatMap(new Splitter()).print();

            environment.execute("data stream test");
        }
    };
}
---------------------------

== Camel-Flink Producer call

[source,java]
-----------------------------------
CamelContext camelContext = new SpringCamelContext(context);

String pattern = "foo";

try {
    ProducerTemplate template = camelContext.createProducerTemplate();
    camelContext.start();
    Long count = template.requestBody("flink:dataSet?dataSet=#myDataSet&dataSetCallback=#countLinesContaining", pattern, Long.class);
    } finally {
        camelContext.stop();
    }
-----------------------------------