[FLINK-23600] [docs] Restructure module configuration docs
This closes #263
diff --git a/docs/content/docs/deployment/module.md b/docs/content/docs/deployment/module.md
deleted file mode 100644
index 0c46956..0000000
--- a/docs/content/docs/deployment/module.md
+++ /dev/null
@@ -1,301 +0,0 @@
----
-title: Module Configuration
-weight: 2
-type: docs
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-# Module Configuration
-
-An application's module configuration contains all necessary runtime information to configure the Stateful Functions runtime for a specific application. It includes the endpoints where functions can be reached along with ingress and egress definitions.
-
-```yaml
-version: "3.0"
-
-module:
- meta:
- type: remote
- spec:
- endpoints:
- meta:
- kind: http
- spec:
- functions: com.example/*
- urlPathTempalte: https://bar.foo.com/{function.name}
- ingresses:
- - ingress:
- meta:
- type: io.statefun.kafka/ingress
- id: com.example/my-ingress
- spec:
- address: kafka-broker:9092
- consumerGroupId: my-consumer-group
- startupPosition:
- type: earliest
- topics:
- - topic: message-topic
- valueType: io.statefun.types/string
- targets:
- - com.example/greeter
- egresses:
- - egress:
- meta:
- type: io.statefun.kafka/egress
- id: example/output-messages
- spec:
- address: kafka-broker:9092
- deliverySemantic:
- type: exactly-once
- transactionTimeout: 15min
-```
-
-## Endpoint Definition
-
-`module.spec.endpoints` declares a list of `endpoint` objects containing all metadata the Stateful Function runtime needs to invoke a function. The following is an example of an endpoint definition for a single function.
-
-```yaml
-endpoints:
- - endpoint:
- meta:
- kind: http
- spec:
- functions: com.example/*
- urlPathTemplate: https://bar.foo.com/{function.name}
-```
-
-In this example, an endpoint for a function within the logical namespace `com.example` is declared.
-The runtime will invoke all functions under this namespace with the endpoint URL template.
-
-## URL Template
-
-The URL template name may contain template parameters that are filled in based on the function's specific type.
-For example, a message sent to message type `com.example/greeter` will be sent to `http://bar.foo.com/greeter`.
-
-```yaml
-endpoints:
- - endpoint:
- meta:
- kind: http
- spec:
- functions: com.example/*
- urlPathTemplate: https://bar.foo.com/{function.name}
-```
-
-Templating parameterization works well with load balancers and service gateways.
-Suppose `http://bar.foo.com` was an [NGINX](https://www.nginx.com/) server, you can use the different paths to physical systems. Users may now deploy some functions on Kubernetes, others AWS Lambda, while others still on physical servers.
-
-{{< img src="/fig/dispatch.png" alt="function dispatch" width="75%" >}}
-
-#### Example Using NGINX
-
-```
-http {
- server {
- listen 80;
- server_name proxy;
-
- location /greeter {
- proxy_pass http://greeter:8000/service;
- }
-
- location /emailsender {
- proxy_pass http://emailsender:8000/service;
- }
- }
-}
-
-events {}
-```
-
-This pattern makes the function's physical deployment transparent to the Stateful Functions runtime.
-They can be deployed, upgraded, rolled back and scaled transparently from the cluster.
-
-### Full Options
-
-#### Protocol
-
-The RPC protocol used to communicate by the runtime to communicate with the function.
-``http`` is currently the only supported value.
-
-```yaml
-endpoint:
- meta:
- kind: http
- spec:
-```
-
-#### Typename
-
-The meta `typename` is the logical type name used to match a function invocation with a physical endpoint.
-Typenames are composed of a namespace (required) and name (optional).
-Endpoints containing a namespace but no name will match all function types in that namespace.
-
-{{< hint info >}}
-It is recommended to have endpoints only specified against a namespace to enable dynamic function registration.
-{{< /hint >}}
-
-```yaml
-endpoint:
- meta:
- kind: http
- spec:
- functions: com.example/*
-```
-
-#### Url Path Template
-
-The `urlPathTemplate` is the physical path to be resolved when an endpoint is matched.
-It may contain templated parameters for dynamic routing.
-
-```yaml
-endpoint:
- meta:
- spec:
- urlPathTemplate: http://bar.foo.com/{function.name}
-```
-
-Supported schemes:
-* ``http``
-* ``https``
-
-Transport via UNIX domain sockets is supported by using the schemes ``http+unix`` or ``https+unix``.
-When using UNIX domain sockets, the endpoint format is: ``http+unix://<socket-file-path>/<serve-url-path>``. For example, ``http+unix:///uds.sock/path/of/url``.
-For example, ``http+unix:///uds.sock/path/of/url``.
-
-#### Max Batch Requests
-
-The maximum number of records that can be processed by a function for a particular address (typename + id) before invoking backpressure on the system. The default value is `1000`.
-
-```yaml
-endpoint:
- meta:
- spec:
- maxNumBatchRequests: 1000
-```
-
-#### Timeouts
-
-Timeouts is an optional object containing various timeouts for a request before a function is considered unreachable.
-
-**Call**
-
-The timeout for a complete function call.
-This configuration spans the entire call: resolving DNS, connecting, writing the request body, server processing, and reading the response body.
-If the call requires redirects or retries all must complete within one timeout period.
-
-```yaml
-endpoint:
- meta:
- spec:
- timeout:
- call: 1 min
-```
-
-**Connect**
-
-The default connect timeout for new connections.
-The connect timeout is applied when connecting a TCP socket to the target host.
-
-```yaml
-endpoint:
- meta:
- spec:
- timeout:
- connect: 10 sec
-```
-
-**Read**
-
-The default read timeout for new connections.
-The read timeout is applied to both the TCP socket and for individual read IO operations.
-
-```yaml
-endpoint:
- meta:
- spec:
- timeout:
- read: 10 sec
-```
-
-**Write**
-
-The default write timeout for new connections.
-The write timeout is applied for individual write IO operations.
-
-```yaml
-endpoint:
- meta:
- spec:
- timeout:
- write: 10 sec
-```
-
-## Ingress
-
-An ingress is an input point where data is consumed from an external system and forwarded to zero or more functions.
-It is defined via an identifier and specification.
-
-An ingress identifier, similar to a function type, uniquely identifies an ingress by specifying its [input type]({{< ref "docs/sdk/appendix#types" >}}), a namespace, and a name.
-
-The spec defines the details of how to connect to the external system, which is specific to each individual I/O module. Each identifier-spec pair is bound to the system inside an stateful function module.
-
-See [IO Modules]{{< ref "docs/io-module/overview" >}} for more information on configuring an ingress.
-
-```yaml
-version: "3.0"
-
-module:
- meta:
- type: remote
- spec:
- ingresses:
- - ingress:
- meta:
- id: example/my-ingress
- type: # ingress type
- spec: # ingress specific configurations
-```
-
-## Egress
-
-An egress is an output point where data is written to an external system and forwarded to zero or more functions.
-It is defined via an identifier and specification.
-
-An egress identifier, similar to a function type, uniquely identifies an egress.
-
-The spec defines the details of how to connect to the external system, which is specific to each individual I/O module.
-Each identifier-spec pair is bound to the system inside a stateful function module.
-
-See [IO Modules]{{< ref "docs/io-module/overview" >}} for more information on configuring an egress.
-
-```yaml
-version: "3.0"
-
-module:
- meta:
- type: remote
- spec:
- egresses:
- - egress:
- meta:
- id: example/my-egress
- type: # egress type
- spec: # egress specific configurations
-```
diff --git a/docs/content/docs/deployment/overview.md b/docs/content/docs/deployment/overview.md
index e6bc3b1..e5e1dd4 100644
--- a/docs/content/docs/deployment/overview.md
+++ b/docs/content/docs/deployment/overview.md
@@ -39,7 +39,7 @@
The provided image allows teams to package their applications with all the necessary runtime dependencies quickly.
The community provides images containing the entire Stateful Functions runtime: `apache/flink-statefun:{{< version >}}`.
-All you need to do is attach your [module configurations]({{< ref "docs/deployment/module" >}}) to the container.
+All you need to do is attach your [module configurations]({{< ref "docs/modules/overview" >}}) to the container.
{{< unstable >}}
{{< hint info >}}
@@ -79,7 +79,7 @@
{{< details "application-module.yaml" >}}
The application function and io module configuration.
-See the full [documentation]({{< ref "docs/deployment/module" >}}) for more details.
+See the full [documentation]({{< ref "docs/modules/overview" >}}) for more details.
```yaml
apiVersion: v1
@@ -91,14 +91,15 @@
app: statefun
data:
module.yaml: |+
- version: "3.0"
- module:
- meta:
- type: remote
- spec:
- endpoints:
- ingresses:
- egresses:
+ kind: ...
+ spec:
+ ...
+ ---
+ kind: ...
+ spec:
+ ...
+ ---
+ ...
```
{{< /details >}}
diff --git a/docs/content/docs/io-module/overview.md b/docs/content/docs/io-module/overview.md
deleted file mode 100644
index b21e826..0000000
--- a/docs/content/docs/io-module/overview.md
+++ /dev/null
@@ -1,96 +0,0 @@
----
-title: 'Overview'
-weight: 1
-type: docs
-aliases:
- - /io-module/
-permalink: /io-module/index.html
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-# I/O Module
-
-Stateful Functions' I/O modules allow functions to receive and send messages to external systems.
-Based on the concept of Ingress (input) and Egress (output) points, and built on top of the Apache Flink® connector ecosystem, I/O modules enable functions to interact with the outside world through the style of message passing.
-
-Commonly used I/O modules are bundled into the runtime by default and can be configured direclty via the applications [module configuration]({{< ref "docs/deployment/module" >}}).
-Additionally, custom connectors for other systems can be [plugged in]({{< ref "docs/io-module/flink-connectors" >}}) to the runtime.
-
-Keep in mind that to use one of these connectors in an application, additional third party components are usually required, e.g. servers for the data stores or message queues.
-
-## Ingress
-
-An Ingress is an input point where data is consumed from an external system and forwarded to zero or more functions.
-It is defined via an ``IngressIdentifier`` and an ``IngressSpec``.
-
-An ingress identifier, similar to a function type, uniquely identifies an ingress by specifying its input type, a namespace, and a name.
-
-The spec defines the details of how to connect to the external system, which is specific to each individual I/O module. Each identifier-spec pair is bound to the system inside an stateful function module.
-```yaml
-version: "3.0"
-
-module:
- meta:
- type: remote
- spec:
- ingresses:
- - ingress:
- meta:
- id: example/user-ingress
- type: # ingress type
- spec: # ingress specific configurations
-```
-
-## Router
-
-A router is a stateless operator that takes each record from an ingress and routes it to zero or more functions.
-Routers are bound to the system via a stateful function module, and unlike other components, an ingress may have any number of routers.
-
-Routers are defined by a list of function types.
-The ``id`` component of the address is pulled from the key associated with each record in its underlying source implementation.
-```yaml
-targets:
- - example-namespace/my-function-1
- - example-namespace/my-function-2
-```
-
-## Egress
-
-Egress is the opposite of ingress; it is a point that takes messages and writes them to external systems.
-Each egress is defined using two components, an ``EgressIdentifier`` and an ``EgressSpec``.
-
-An egress identifier uniquely identifies an egress based on a namespace, name, and producing type.
-An egress spec defines the details of how to connect to the external system, the details are specific to each individual I/O module.
-Each identifier-spec pair is bound to the system inside a Stateful Functions module.
-
-```yaml
-version: "3.0"
-
-module:
- meta:
- type: remote
- spec:
- egresses:
- - egress:
- meta:
- id: example/user-egress
- type: # egress type
- spec: # egress specific configurations
-```
diff --git a/docs/content/docs/io-module/_index.md b/docs/content/docs/modules/_index.md
similarity index 97%
rename from docs/content/docs/io-module/_index.md
rename to docs/content/docs/modules/_index.md
index aacee54..5b98067 100644
--- a/docs/content/docs/io-module/_index.md
+++ b/docs/content/docs/modules/_index.md
@@ -1,5 +1,5 @@
---
-title: I/O Modules
+title: Modules
icon: <i class="fa fa-random title maindish" aria-hidden="true"></i>
bold: true
bookCollapseSection: true
diff --git a/docs/content/docs/deployment/embedded.md b/docs/content/docs/modules/embedded.md
similarity index 96%
rename from docs/content/docs/deployment/embedded.md
rename to docs/content/docs/modules/embedded.md
index de988dd..3ac92e6 100644
--- a/docs/content/docs/deployment/embedded.md
+++ b/docs/content/docs/modules/embedded.md
@@ -1,6 +1,6 @@
---
-title: Embedded Module
-weight: 3
+title: 'Embedded Modules'
+weight: 5
type: docs
---
<!--
@@ -25,7 +25,7 @@
# Embedded Module Configuration
Embedded modules allow users to load code into the Stateful Functions runtime that is executed directly within the cluster.
-This is usually to allow plugging in [custom ingress and egress implementations]({{< ref "docs/io-module/flink-connectors">}}).
+This is usually to allow plugging in [custom ingress and egress implementations]({{< ref "docs/modules/io/flink-connectors">}}).
Additionally, and embedded module may include [embedded functions]({{< ref "docs/sdk/flink-datastream#embedded-functions" >}}) that run within the cluster.
Embedded modules should be used with care, they cannot be deployed or scaled without downtime and can effect the performance and stability of the entire cluster.
diff --git a/docs/content/docs/modules/http-endpoint.md b/docs/content/docs/modules/http-endpoint.md
new file mode 100644
index 0000000..5e7e915
--- /dev/null
+++ b/docs/content/docs/modules/http-endpoint.md
@@ -0,0 +1,207 @@
+---
+title: 'HTTP Function Endpoint'
+weight: 2
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# HTTP Function Endpoint
+
+An HTTP Function Endpoint component defines the endpoint URL that the Stateful Functions runtime should connect to for
+invoking a given function, or for a more dynamic deployment, functions within a specified namespace.
+
+Below is an example of an HTTP endpoint definition in an application's module configuration:
+
+{{< tabs "8951ef0a-cdd4-40d1-bda8-dec1299aaf41" >}}
+{{< tab "v2 (latest)" >}}
+```yaml
+kind: io.statefun.endpoints.v2/http
+spec:
+ functions: com.example/*
+ urlPathTemplate: https://bar.foo.com:8080/{function.name}
+ transport:
+ timeouts:
+ call: 1 min
+ read: 10 sec
+ write: 10 sec
+```
+{{< /tab >}}
+{{< tab "v1" >}}
+```yaml
+kind: io.statefun.endpoints.v1/http
+spec:
+ functions: com.example/*
+ urlPathTemplate: https://bar.foo.com:8080/{function.name}
+ timeouts:
+ call: 1 min
+ read: 10 sec
+ write: 10 sec
+```
+{{< /tab >}}
+{{< /tabs >}}
+
+In this example, an endpoint for a function within the logical namespace `com.example` is declared.
+The runtime will invoke all functions under this namespace with the endpoint URL template.
+
+### URL Template
+
+The URL template name may contain template parameters filled in dynamically based on the function's specific type.
+In the example below, a message sent to message type `com.example/greeter` will be sent to `http://bar.foo.com/greeter`.
+
+```yaml
+spec:
+ functions: com.example/*
+ urlPathTemplate: https://bar.foo.com/{function.name}
+```
+
+Templating parameterization works well with load balancers and service gateways.
+Suppose `http://bar.foo.com` was an [NGINX](https://www.nginx.com/) server; you can use the different paths to physical systems. Users may now deploy some functions on Kubernetes, others AWS Lambda, while others are still on physical servers.
+
+{{< img src="/fig/dispatch.png" alt="function dispatch" width="75%" >}}
+
+### Transport
+
+Switching between different transport clients is supported since `v2` of HTTP function endpoint definitions.
+
+The transport client to use is specified using `spec.transport.type`. If not specified, by default, Stateful Functions uses [OkHttp](https://square.github.io/okhttp/).
+
+All fields under `spec.transport` is used as the properties to configure the transport client. For example, the example below configures various timeout settings for the default `OkHttp` transport:
+
+```yaml
+spec:
+ transport:
+ timeouts:
+ call: 1 min
+ read: 30 sec
+ write: 20 sec
+```
+
+#### Asynchronous HTTP transport (Beta)
+
+Alternatively, Stateful Functions also ships a transport option based on asynchronous non-blocking IO, implemented with [Netty](https://netty.io/).
+This transport enables much higher resource utilization, higher throughput, and lower remote function invocation latency.
+
+Below is a complete example of the `transport` section of an HTTP function endpoint definition, if you want to use this transport type:
+
+```yaml
+spec:
+ transport:
+ type: io.statefun.transports.v1/async
+ call: 2m
+ connect: 20s
+ pool_ttl: 15s
+ pool_size: 1024
+ payload_max_bytes: 33554432
+```
+
+Please see the full spec options below for a description of each property.
+
+## Full Spec Options
+
+#### Target functions (or function)
+
+The meta `typename` is the logical type name used to match a function invocation with a physical endpoint.
+Typenames are composed of a namespace (required) and name (optional).
+Endpoints containing a namespace but no name will match all function types in that namespace.
+
+{{< hint info >}}
+It is recommended to have endpoints only specified against a namespace to enable dynamic function registration.
+{{< /hint >}}
+
+```yaml
+spec:
+ functions: com.example/*
+```
+
+#### Url Path Template
+
+The `urlPathTemplate` is the physical path to be resolved when an endpoint is matched.
+It may contain templated parameters for dynamic routing.
+
+```yaml
+spec:
+ urlPathTemplate: http://bar.foo.com/{function.name}
+```
+
+Supported schemes:
+* ``http``
+* ``https``
+
+Transport via UNIX domain sockets is supported by using the schemes ``http+unix`` or ``https+unix``.
+When using UNIX domain sockets, the endpoint format is: ``http+unix://<socket-file-path>/<serve-url-path>``. For example, ``http+unix:///uds.sock/path/of/url``.
+For example, ``http+unix:///uds.sock/path/of/url``.
+
+#### Max Batch Requests
+
+The maximum number of records that can be processed by a function for a particular address (typename + id) before invoking backpressure on the system. The default value is `1000`.
+
+```yaml
+spec:
+ maxNumBatchRequests: 1000
+```
+
+#### Transport
+
+The transport client type to use for sending requests to the function. If not specified, by default, Stateful Functions uses [OkHttp](https://square.github.io/okhttp/).
+
+Below is a full example for configuring the default transport:
+
+```yaml
+spec:
+ transport:
+ timeouts:
+ call: 2m
+ connect: 20s
+ read: 10s
+ write: 10s
+```
+
+* `call`: The timeout for a complete function call. This configuration spans the entire call: resolving DNS, connecting,
+ writing the request body, server processing, and reading the response body. If the call requires redirects or retries
+ all must complete within one timeout period. Default value is 1 minute.
+* `connect`: The default connect timeout for new connections. The connect timeout is applied when connecting a TCP socket to the target host. Default value is 10 seconds.
+* `read`: The default read timeout for new connections. The read timeout is applied to both the TCP socket and for individual read IO operations. Default value is 10 seconds.
+* `write`: The default write timeout for new connections. The write timeout is applied for individual write IO operations. Default value is 10 seconds.
+
+Alternatively, a transport client based on asynchronous non-blocking IO is supported:
+
+```yaml
+spec:
+ transport:
+ type: io.statefun.transports.v1/async
+ call: 2m
+ connect: 20s
+ pool_ttl: 15s
+ pool_size: 1024
+ payload_max_bytes: 33554432
+```
+
+* `call`: total duration of a single request (including retries, and backoffs). After this duration, the call is considered failed.
+* `connect`: the total amount of time to wait for a successful TCP connection. After that amount of time, an attempt is considered failed, and if the total call time has not elapsed, an additional attempt will be scheduled (after a backoff).
+* `pool_ttl`: the amount of time a connection will live in the connection pool. Set to 0 to disable, otherwise the connection will be evicted from the pool after (approximately) that time. If a connection is evicted while it is serving a request, that connection will be only marked for eviction and will be dropped from the pool once the request returns.
+* `pool_size`: the maximum pool size.
+* `payload_max_bytes`: the maximum size for a request or response payload size. The default is set to 32MB.
+
+{{< hint info >}}
+We highly recommend setting `statefun.async.max-per-task` to a much higher value (see [Configurations]({{< ref "docs/deployment/configurations">}}))
+when using the asynchronous transport type. This configuration dictates the maximum amount of in-flight requests sent to
+the function before applying back pressure. The default value of this configuration is `1024`, which makes sense only for
+the default threaded blocking `OkHttp` transport. For the asynchronous transport, a much higher value is recommended.
+{{< /hint >}}
\ No newline at end of file
diff --git a/docs/content/docs/io-module/_index.md b/docs/content/docs/modules/io/_index.md
similarity index 87%
copy from docs/content/docs/io-module/_index.md
copy to docs/content/docs/modules/io/_index.md
index aacee54..16a9e62 100644
--- a/docs/content/docs/io-module/_index.md
+++ b/docs/content/docs/modules/io/_index.md
@@ -1,9 +1,9 @@
---
-title: I/O Modules
-icon: <i class="fa fa-random title maindish" aria-hidden="true"></i>
-bold: true
+title: IO Components
+weight: 3
bookCollapseSection: true
-weight: 4
+aliases:
+ - /modules/io/
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
diff --git a/docs/content/docs/io-module/apache-kafka.md b/docs/content/docs/modules/io/apache-kafka.md
similarity index 82%
rename from docs/content/docs/io-module/apache-kafka.md
rename to docs/content/docs/modules/io/apache-kafka.md
index f331dc2..8a2c643 100644
--- a/docs/content/docs/io-module/apache-kafka.md
+++ b/docs/content/docs/modules/io/apache-kafka.md
@@ -3,7 +3,7 @@
weight: 2
type: docs
aliases:
-- /io-module/apache-kafka.html
+- /modules/io/apache-kafka.html
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -28,35 +28,30 @@
Stateful Functions offers an Apache Kafka I/O Module for reading from and writing to Kafka topics.
It is based on Apache Flink's universal [Kafka connector](https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html) and provides exactly-once processing semantics.
-Kafka is configured in the [module specification]({{< ref "docs/deployment/module" >}}) of your application.
+Kafka is configured in the [module specification]({{< ref "docs/modules/overview" >}}) of your application.
## Kafka Ingress Spec
A Kafka ingress defines an input point that reads records from one or more topics.
+{{< tabs "8951ef0a-cdd4-40d1-bda8-dec1299aaf42" >}}
+{{< tab "v1 (latest)" >}}
```yaml
-version: "3.0"
-
-module:
- meta:
- type: remote
+kind: io.statefun.kafka.v1/ingress
spec:
- ingresses:
- - ingress:
- meta:
- type: io.statefun.kafka/ingress
- id: com.example/users
- spec:
- address: kafka-broker:9092
- consumerGroupId: my-consumer-group
- startupPosition:
- type: earliest
- topics:
- - topic: messages-1
- valueType: com.example/User
- targets:
- - com.example.fns/greeter
+ id: com.example/users
+ address: kafka-broker:9092
+ consumerGroupId: my-consumer-group
+ startupPosition:
+ type: earliest
+ topics:
+ - topic: messages-1
+ valueType: com.example/User
+ targets:
+ - com.example.fns/greeter
```
+{{< /tab >}}
+{{< /tabs >}}
The ingress also accepts properties to directly configure the Kafka client, using ``ingress.spec.properties``.
Please refer to the Kafka [consumer configuration](https://docs.confluent.io/current/installation/configuration/consumer-configs.html) documentation for the full list of available properties.
@@ -124,26 +119,21 @@
A Kafka egress defines an input point where functions can write out records to one or more topics.
+{{< tabs "8951ef0a-cdd4-40d1-bda8-dec1299aaf41" >}}
+{{< tab "v1 (latest)" >}}
```yaml
-version: "3.0"
-
-module:
- meta:
- type: remote
+kind: io.statefun.kafka.v1/egress
spec:
- egresses:
- - egress:
- meta:
- type: io.statefun.kafka/egress
- id: example/output-messages
- spec:
- address: kafka-broker:9092
- deliverySemantic:
- type: exactly-once
- transactionTimeout: 15min
- properties:
- - foo.config: bar
+ id: com.example/users
+ address: kafka-broker:9092
+ deliverySemantic:
+ type: exactly-once
+ transactionTimeout: 15min
+ properties:
+ - foo.config: bar
```
+{{< /tab >}}
+{{< /tabs >}}
Please refer to the Kafka [producer configuration](https://docs.confluent.io/current/installation/configuration/producer-configs.html) documentation for the full list of available properties.
diff --git a/docs/content/docs/io-module/aws-kinesis.md b/docs/content/docs/modules/io/aws-kinesis.md
similarity index 75%
rename from docs/content/docs/io-module/aws-kinesis.md
rename to docs/content/docs/modules/io/aws-kinesis.md
index c8947c1..ebc44aa 100644
--- a/docs/content/docs/io-module/aws-kinesis.md
+++ b/docs/content/docs/modules/io/aws-kinesis.md
@@ -3,7 +3,7 @@
weight: 3
type: docs
aliases:
-- /io-module/aws-kinesis.html
+- /modules/io/aws-kinesis.html
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -29,42 +29,35 @@
Stateful Functions offers an AWS Kinesis I/O Module for reading from and writing to Kinesis streams.
It is based on Apache Flink's [Kinesis connector](https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/connectors/kinesis.html).
-Kinesis is configured in the [module specification]({{< ref "docs/deployment/module" >}}) of your application.
+Kinesis is configured in the [module specification]({{< ref "docs/modules/overview" >}}) of your application.
## Kinesis Ingress Spec
A Kinesis ingress defines an input point that reads records from one or more streams.
+{{< tabs "8951ef0a-cdd4-40d1-bda8-dec1299aaf42" >}}
+{{< tab "v1 (latest)" >}}
```yaml
-version: "3.0"
-
-module:
- meta:
- type: remote
- spec:
- ingresses:
- - ingress:
- meta:
- type: io.statefun.kinesis/ingress
- id: com.example/users
- spec:
- awsRegion:
- type: specific
- id: eu-west-1
- startupPosition:
- type: latest
- streams:
- - stream: user-stream
- valueType: com.example/User
- targets:
- - com.example.fn/greeter
- clientConfigProperties:
- - SocketTimeout: 9999
- - MaxConnections: 15
- type: statefun.kinesis.io/routable-protobuf-ingress
- id: example-namespace/messages
+kind: io.statefun.kinesis.v1/ingress
+spec:
+ id: com.example/users
+ awsRegion:
+ type: specific
+ id: eu-west-1
+ startupPosition:
+ type: latest
+ streams:
+ - stream: user-stream
+ valueType: com.example/User
+ targets:
+ - com.example.fn/greeter
+ clientConfigProperties:
+ - SocketTimeout: 9999
+ - MaxConnections: 15
```
+{{< /tab >}}
+{{< /tabs >}}
Please refer to the Kinesis [client configuration](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/ClientConfiguration.html) documentation for the full list of available properties.
Note that configuration passed using named methods will have higher precedence and overwrite their respective settings in the provided properties.
@@ -105,29 +98,24 @@
A Kinesis egress defines an input point where functions can write out records to one or more streams.
+{{< tabs "8951ef0a-cdd4-40d1-bda8-dec1299aaf41" >}}
+{{< tab "v1 (latest)" >}}
```yaml
-version: "3.0"
-
-module:
- meta:
- type: remote
- spec:
- egresses:
- - egress:
- meta:
- type: io.statefun.kinesis/egress
- id: com.example/out
- spec:
- awsRegion:
- type: specific
- id: eu-west-1
- awsCredentials:
- type: default
- maxOutstandingRecords: 9999
- clientConfigProperties:
- - ThreadingModel: POOLED
- - ThreadPoolSize: 10
+kind: io.statefun.kinesis.v1/egress
+spec:
+ id: com.example/out
+ awsRegion:
+ type: specific
+ id: eu-west-1
+ awsCredentials:
+ type: default
+ maxOutstandingRecords: 9999
+ clientConfigProperties:
+ - ThreadingModel: POOLED
+ - ThreadPoolSize: 10
```
+{{< /tab >}}
+{{< /tabs >}}
Please refer to the Kinesis [producer default configuration properties](https://github.com/awslabs/amazon-kinesis-producer/blob/master/java/amazon-kinesis-producer-sample/default_config.properties) documentation for the full list of available properties.
diff --git a/docs/content/docs/io-module/flink-connectors.md b/docs/content/docs/modules/io/flink-connectors.md
similarity index 98%
rename from docs/content/docs/io-module/flink-connectors.md
rename to docs/content/docs/modules/io/flink-connectors.md
index 330cc4a..5f2699a 100644
--- a/docs/content/docs/io-module/flink-connectors.md
+++ b/docs/content/docs/modules/io/flink-connectors.md
@@ -3,7 +3,7 @@
weight: 4
type: docs
aliases:
- - /io-module/flink-connectors.html
+ - /modules/io/flink-connectors.html
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -29,7 +29,7 @@
The source-sink I/O module allows you to plug in existing, or custom, Flink connectors that are not already integrated into a dedicated I/O module.
Please see the official Apache Flink documentation for a full list of [available connectors](https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/connectors/) as well as details on how to build your own.
-Connectors can be plugged into the runtime via an [embedded module]({{< ref "docs/deployment/embedded" >}})
+Connectors can be plugged into the runtime via an [embedded module]({{< ref "docs/modules/embedded" >}})
## Dependency
diff --git a/docs/content/docs/modules/io/overview.md b/docs/content/docs/modules/io/overview.md
new file mode 100644
index 0000000..a49462e
--- /dev/null
+++ b/docs/content/docs/modules/io/overview.md
@@ -0,0 +1,36 @@
+---
+title: 'Overview'
+weight: 1
+type: docs
+aliases:
+ - /modules/io/
+permalink: /modules/io/index.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# I/O Module
+
+Stateful Functions' I/O modules allow functions to receive and send messages to external systems.
+Based on the concept of Ingress (input) and Egress (output) points, and built on top of the Apache Flink® connector ecosystem, I/O modules enable functions to interact with the outside world through the style of message passing.
+
+Commonly used I/O modules are bundled into the runtime by default and can be configured directly via the applications [module configuration]({{< ref "docs/modules/overview" >}}).
+Additionally, custom connectors for other systems can be [plugged in]({{< ref "docs/modules/io/flink-connectors" >}}) to the runtime.
+
+Remember, to use one of these connectors in an application, third-party components are usually required, e.g., servers for the data stores or message queues.
diff --git a/docs/content/docs/modules/overview.md b/docs/content/docs/modules/overview.md
new file mode 100644
index 0000000..1cc3492
--- /dev/null
+++ b/docs/content/docs/modules/overview.md
@@ -0,0 +1,64 @@
+---
+title: 'Overview'
+weight: 1
+type: docs
+aliases:
+ - /modules/
+permalink: /modules/index.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Application Modules
+
+An application module consists of multiple [components]({{< ref "docs/concepts/application-building-blocks" >}})
+that take part in a StateFun application. It includes the endpoints where the runtime can reach functions, along with ingress and egress definitions.
+
+Modules are defined using a YAML file. For example, below is a module that defines an HTTP function endpoint as well as
+a Kafka ingress and egress:
+
+```yaml
+kind: io.statefun.endpoints.v2/http
+spec:
+ functions: com.example/*
+ urlPathTemplate: https://bar.foo.com/{function.name}
+---
+kind: io.statefun.kafka.v1/ingress
+spec:
+ id: com.example/my-ingress
+ address: kafka-broker:9092
+ consumerGroupId: my-consumer-group
+ topics:
+ - topic: message-topic
+ valueType: io.statefun.types/string
+ targets:
+ - com.example/greeter
+---
+kind: io.statefun.kafka.v1/egress
+spec:
+ id: com.example/my-egress
+ address: kafka-broker:9092
+ deliverySemantic:
+ type: exactly-once
+ transactionTimeout: 15min
+---
+```
+
+A module YAML file can contain multiple YAML documents, separated by `---`, each representing a component to be included in the application.
+Each component is defined by a kind typename string and a spec object containing the component's properties.
diff --git a/docs/content/docs/sdk/flink-datastream.md b/docs/content/docs/sdk/flink-datastream.md
index f222487..ba5d217 100644
--- a/docs/content/docs/sdk/flink-datastream.md
+++ b/docs/content/docs/sdk/flink-datastream.md
@@ -104,7 +104,7 @@
As you can see, instead of binding functions, ingresses, and egresses through modules as you would with a typical Stateful
Functions application, you bind them directly to the ``DataStream`` job using a ``StatefulFunctionDataStreamBuilder``:
-* Remote functions are bound using the `withRequestReplyRemoteFunction` method. [Specification of the remote function]({{< ref "docs/deployment/module" >}})
+* Remote functions are bound using the `withRequestReplyRemoteFunction` method. [Specification of the remote function]({{< ref "docs/modules/http-endpoint" >}})
such as service endpoint and various connection configurations can be set using the provided ``RequestReplyFunctionBuilder``.
* [Embedded functions](#embedded-functions) are bound using ``withFunctionProvider``.
* Egress identifiers used by functions need to be bound with the `withEgressId` method.
diff --git a/docs/content/docs/sdk/golang.md b/docs/content/docs/sdk/golang.md
index e88a3c2..f2ef316 100644
--- a/docs/content/docs/sdk/golang.md
+++ b/docs/content/docs/sdk/golang.md
@@ -359,4 +359,4 @@
## Next Steps
-Keep learning with information on setting up [I/O modules]({{< ref "docs/io-module/overview" >}}) and configuring the [Stateful Functions runtime]({{< ref "docs/deployment/overview" >}}).
+Keep learning with information on setting up [I/O components]({{< ref "docs/modules/io/overview" >}}) and configuring the [Stateful Functions runtime]({{< ref "docs/deployment/overview" >}}).
diff --git a/docs/content/docs/sdk/java.md b/docs/content/docs/sdk/java.md
index bfa1ec7..ccb0ed0 100644
--- a/docs/content/docs/sdk/java.md
+++ b/docs/content/docs/sdk/java.md
@@ -504,4 +504,4 @@
## Next Steps
-Keep learning with information on setting up [I/O modules]({{< ref "docs/io-module/overview" >}}) and configuring the [Stateful Functions runtime]({{< ref "docs/deployment/overview" >}}).
+Keep learning with information on setting up [I/O components]({{< ref "docs/modules/io/overview" >}}) and configuring the [Stateful Functions runtime]({{< ref "docs/deployment/overview" >}}).
diff --git a/docs/content/docs/sdk/python.md b/docs/content/docs/sdk/python.md
index c9cde7d..e67ad96 100644
--- a/docs/content/docs/sdk/python.md
+++ b/docs/content/docs/sdk/python.md
@@ -306,4 +306,4 @@
## Next Steps
-Keep learning with information on setting up [I/O modules]({{< ref "docs/io-module/overview" >}}) and configuring the [Stateful Functions runtime]({{< ref "docs/deployment/overview" >}}).
+Keep learning with information on setting up [I/O components]({{< ref "docs/modules/io/overview" >}}) and configuring the [Stateful Functions runtime]({{< ref "docs/deployment/overview" >}}).
