[ISSUE #220] Add HTTP Sink Connector Document (#221)
* docs: add HTTP Sink Connector Chinese docs
* docs: add HTTP Sink Connector English docs
* fix:update something
* fix:update something
diff --git a/docs/design-document/03-connect/02-http-connector.md b/docs/design-document/03-connect/02-http-connector.md
index 3aeeb5b..b62b776 100644
--- a/docs/design-document/03-connect/02-http-connector.md
+++ b/docs/design-document/03-connect/02-http-connector.md
@@ -5,12 +5,14 @@
### Configuration
Before using HTTP source connector, you need to configure the server.
+
- Please configure `sourceEnable` to `true` in `/resource/server-config.yml` to enable source functionality.
-- Please configure the source connector in `/resource/source-config.yml`, only the configuration under `connectorConfig` is described here:
- - `connectorName`, name of the connector.
- - (required) `path`, path of the API.
- - (required) `port`, port of the API.
- - `idleTimeout`, idle TCP connection timeout in seconds. A connection will timeout and be closed if no data is received nor sent within the `idleTimeout` seconds. The default is 0, which means don't timeout.
+- Please configure the source connector in `/resource/source-config.yml`, only the configuration under `connectorConfig`
+ is described here:
+ - `connectorName`, name of the connector.
+ - (required) `path`, path of the API.
+ - (required) `port`, port of the API.
+ - `idleTimeout`, idle TCP connection timeout in seconds. A connection will timeout and be closed if no data is received nor sent within the `idleTimeout` seconds. The default is 0, which means don't timeout.
### Startup
@@ -25,15 +27,16 @@
```yaml
connectorConfig:
- connectorName: httpSource
- path: /test
- port: 3755
- idleTimeout: 5
+ connectorName: httpSource
+ path: /test
+ port: 3755
+ idleTimeout: 5
```
The above example configures a URL `http://localhost:3755/test` in `source-config.yml`.
-You can send messages in `binary` mode or `structured` mode as specified in [cloudevent-spec](https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/bindings/http-protocol-binding.md).
+You can send messages in `binary` mode or `structured` mode as specified
+in [cloudevent-spec](https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/bindings/http-protocol-binding.md).
Here are two examples:
@@ -65,3 +68,147 @@
"data": "testdata"
}'
```
+
+## HTTP Sink Connector
+
+The HTTP sink connector has two modes: common and webhook.
+
+First of all, either mode has the basic function of sinkConnector, which is to send a message to the target HTTP server.
+When the response is received from the HTTP server, the common mode only cares about the status code of the response to
+determine whether it was sent successfully, while the webook mode not only cares about whether it was sent successfully
+or not, but also stores the data of the response (called the callback data) and provides callback data exposure services
+to the outside world.
+
+The specific differences between the two are as follows:
+
+- Default Idle TCP Connection Timeout
+
+ The default value is 5000ms for common mode and 15000ms for webhook mode.
+
+- Processing of response results
+
+ The common mode only cares about the status code of the response, while the webhook mode will also store the callback
+ data and provide callback data exposure services to the public.
+
+### Configuration
+
+Before using the HTTP sink connector, you need to configure the sink.
+
+- Please configure `sinkEnable` to true in `/resource/server-config.yml` to enable sink function.
+
+- Please configure the sink connector in `/resource/sink-config.yml`, only the configuration under `connectorConfig` is
+ described here:
+
+ The simplest configuration of the common mode:
+
+ ```yaml
+ connectorConfig:
+ connectorName: httpSink
+ urls:
+ - http://127.0.0.1:8987/test
+ ```
+
+ The simplest configuration for webhook mode:
+
+ ```yaml
+ connectorConfig:
+ connectorName: httpSink
+ urls:
+ - http://127.0.0.1:8987/test
+ webhookConfig:
+ activate: true
+ port: 8988
+ ```
+
+All configurations for the HTTP sink connector are listed below:
+
+| Field | Type | Required or not | Clarification | Remark |
+|---------------------------------|---------|--------------------|------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| connectorName | String | yes | Connector Name | |
+| urls | List | yes | List of URLs to receive messages | The default port is 80 for HTTP protocol and 443 for HTTPS protocol<br />The legal URL format are:<br />http://127.0.0.1:8987/test<br />http://127.0.0.1/test<br />https://example.com:4943/test<br/>https://example.com/test<br/> |
+| keepAlive | Boolean | no | Whether to use HTTP persistent connection | Default: true |
+| keepAliveTimeout | Integer | no | HTTP persistent connection timeout | Unit: ms,default: 60000 |
+| connectionTimeout | Integer | no | TCP connection timeout | Unit: ms,default: 5000 |
+| idleTimeout | Integer | no | TCP idle timeout | Unit: ms,default: 5000(common), 15000(webhook) |
+| maxConnectionPoolSize | Integer | no | Maximum number of HTTP connections for the client | Default: 5 |
+| retryConfig | Object | | Configuration of the retry mechanism | |
+| retryConfig.maxRetries | Integer | no | Maximum number of retries | Does not include first attempt, default: 2 |
+| retryConfig.interval | Integer | no | Retry interval | Unit: ms, default: 2000 |
+| retryConfig.retryOnNonSuccess | Boolean | no | Whether to retry requests that receive a non-2xx response | Default: false, retries only network level error requests |
+| webhookConfig | Object | | Configuration of Webhook Mode | |
+| webhookConfig.activate | Boolean | no | Whether to enable Webhook mode | Default: false |
+| webhookConfig.exportPath | String | no | Callback data exposure path | Default: /export<br/>The request method is fixed to GET<br/>Request Parameters:<br />type：peek (default value, get data)/poll (get and delete data)<br />pageNum: must be greater than 0 when type is peek, invalid when type is poll<br/>pageSize: must be greater than 0 |
+| webhookConfig.port | Integer | Yes (when enabled) | Callback data exposure port | |
+| webhookConfig.serverIdleTimeout | Integer | no | TCP idle timeout length for servers exposing callback data | Unit: ms, default: 5000 |
+| webhookConfig.maxStorageSize | Integer | no | Store the maximum value of the callback data queue | Default: 5000 |
+
+### Startup
+
+1. start EventMesh Runtime
+2. start eventmesh-connector-http
+
+Once that's done, send a message to the EventMesh, and then your message will happen to the target HTTP server via an
+HTTP request. If webhook mode is enabled, an HTTP server will also be opened at the same time to provide callback data
+exposure to the outside world.
+
+### Data Format
+
+The data format of the message sent by the HTTP sink connector is as follows:
+
+- type: consists of connectorName, protocol, and mode.
+- time: the time the request was sent
+- uuid: request unique identifier, one-to-one with the uuid of the callback data structure in webhook mode
+- eventId: event ID, composed of type and offset
+- data：actual data to be sent
+
+```json
+{
+ "type": "httpSink.http.webhook",
+ "time": "2024-05-09T22:27:02.028",
+ "uuid": "951d88ee-2503-4df3-a1dd-6e4a6a1c3d3b",
+ "eventId": "httpSink.http.webhook-0",
+ "data": ${ConnectRecord}
+}
+```
+
+When webhook mode is enabled, an HTTP server is also enabled to provide callback data exposure to the outside world. The
+format of the callback data is as follows:
+
+- pageNum: current page number
+- pageSize: size per page
+- pageItems: the callback data array obtained
+- The meaning of the fields of pageItem is as follows:
+ - data: callback data returned from the callback server
+ - metadata: metadata, the fields have the following meanings:
+ - url: the URL to get the callback data
+ - code: the status code of the response to the request callback data, or -1 if a network level error occurred (
+ response could not be received).
+ - message: response information from the request callback data, or exception information if a network level
+ error occurred (response could not be received).
+ - uuid: a unique request identifier for the callback data, corresponding to the uuid carried in the message sent
+ above.
+ - receivedTime: response time for requesting callback data
+ - retryNum: number of retries
+ - retriedBy: record the uuid of the request being retried if a retry is sent, otherwise null
+
+```json
+{
+ "pageNum": 1,
+ "pageSize": 10,
+ "pageItems": [
+ {
+ "data": "callbackData",
+ "metadata": {
+ "url": "http://127.0.0.1:8987/test",
+ "code": 200,
+ "message": "OK",
+ "receivedTime": "2024-05-09 22:53:21.556",
+ "uuid": "fba29061-1a1f-4482-9c83-43ba4e0bcf3f",
+ "retriedBy": null,
+ "retryNum": 0
+ }
+ }
+ ]
+}
+```
+
diff --git a/docs/roadmap.md b/docs/roadmap.md
index 6fb92e3..9ac2e40 100644
--- a/docs/roadmap.md
+++ b/docs/roadmap.md
@@ -57,7 +57,7 @@
| [Kafka](https://github.com/apache/eventmesh/tree/master/eventmesh-connectors/eventmesh-connector-kafka) | ✅ | ✅ |
| [Pulsar](https://github.com/apache/eventmesh/tree/master/eventmesh-connectors/eventmesh-connector-pulsar) | ✅ | ✅ |
| [RabbitMQ](./design-document/connect/rabbitmq-connector) | ✅ | ✅ |
-| [HTTP](./design-document/connect/http-connector) | ✅ | ⬜ |
+| [HTTP](./design-document/connect/http-connector) | ✅ | ✅ |
| [JDBC](https://github.com/apache/eventmesh/tree/master/eventmesh-connectors/eventmesh-connector-jdbc) | ⬜ | ✅ |
| [Spring](https://github.com/apache/eventmesh/tree/master/eventmesh-connectors/eventmesh-connector-spring) | ✅ | ✅ |
| [OpenFunction](https://github.com/apache/eventmesh/tree/master/eventmesh-connectors/eventmesh-connector-openfunction) | ✅ | ✅ |
diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/design-document/03-connect/02-http-connector.md b/i18n/zh/docusaurus-plugin-content-docs/current/design-document/03-connect/02-http-connector.md
index 95287d5..710e47b 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/current/design-document/03-connect/02-http-connector.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/current/design-document/03-connect/02-http-connector.md
@@ -25,10 +25,10 @@
```yaml
connectorConfig:
- connectorName: httpSource
- path: /test
- port: 3755
- idleTimeout: 5
+ connectorName: httpSource
+ path: /test
+ port: 3755
+ idleTimeout: 5
```
上述的例子在`source-config.yml`中配置了一个 URL `http://localhost:3755/test`.
@@ -64,4 +64,135 @@
"datacontenttype":"text/plain",
"data": "testdata"
}'
-```
\ No newline at end of file
+```
+
+## HTTP Sink Connector
+
+HTTP sink connector 拥有两种模式：common 和 webhook。
+
+首先，无论是哪种模式，都具备sinkConnector的基本功能，即将消息发送给目标HTTP服务器。当收到HTTP服务器的响应时，common模式只关心响应的状态码，以判断是否发送成功，而webook模式除了关心是否发送成功以外，还将响应的数据（称为回调数据）进行存储，并对外提供回调数据暴露服务。两者的具体差别如下：
+
+- 默认空闲 TCP 连接超时时间
+
+ common模式的默认值为5000ms，webhook模式则为15000ms。
+
+- 响应结果的处理
+
+ common模式只关心响应的状态码，而webhook模式还会存储回调数据，并对外提供回调数据暴露服务。
+
+### 配置
+
+使用 HTTP sink connector 前，需要进行 sink 的配置。
+
+- 请在 `/resource/server-config.yml` 中配置 `sinkEnable`为`true` 以开启sink 功能。
+
+- 请在 `/resource/sink-config.yml`中配置 sink connector,在此仅说明 `connectorConfig` 下的配置：
+
+ common模式的最简配置：
+
+ ```yaml
+ connectorConfig:
+ connectorName: httpSink
+ urls:
+ - http://127.0.0.1:8987/test
+ ```
+
+ webhook模式的最简配置：
+
+ ```yaml
+ connectorConfig:
+ connectorName: httpSink
+ urls:
+ - http://127.0.0.1:8987/test
+ webhookConfig:
+ activate: true
+ port: 8988
+ ```
+
+HTTP sink connector 的所有配置如下：
+
+| 属性 | 类型 | 是否必填 | 说明 | 备注 |
+|---------------------------------|---------|--------|----------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| connectorName | String | 是 | connector名称 | |
+| urls | List | 是 | 接收消息的URL列表 | HTTP协议默认端口为80,HTTPS协议则为443<br />合法URL格式为：<br />http://127.0.0.1:8987/test<br />http://127.0.0.1/test<br />https://example.com:4943/test<br/>https://example.com/test<br/> |
+| keepAlive | Boolean | 否 | 是否使用HTTP持久连接 | 默认：true |
+| keepAliveTimeout | Integer | 否 | HTTP持久连接超时时长 | 单位：ms，默认：60000 |
+| connectionTimeout | Integer | 否 | TCP连接超时时长 | 单位：ms，默认：5000 |
+| idleTimeout | Integer | 否 | TCP空闲超时时长 | 单位：ms，默认：5000（common），15000（webhook） |
+| maxConnectionPoolSize | Integer | 否 | 客户端的最大HTTP连接数 | 默认：5 |
+| retryConfig | Object | | 重试机制的相关配置 | |
+| retryConfig.maxRetries | Integer | 否 | 最大重试次数 | 不包含首次尝试，默认：2 |
+| retryConfig.interval | Integer | 否 | 重试间隔 | 单位：ms，默认：2000 |
+| retryConfig.retryOnNonSuccess | Boolean | 否 | 是否重试收到非2xx响应的请求 | 默认：false，仅重试网络层面的错误请求 |
+| webhookConfig | Object | | Webhook模式的相关配置 | |
+| webhookConfig.activate | Boolean | 否 | 是否启用Webhook模式 | 默认：false |
+| webhookConfig.exportPath | String | 否 | 回调数据暴露路径 | 默认：/export<br/>请求方式固定为GET<br/>请求参数有<br />type：peek(默认值，获取数据)/poll(获取并删除数据)<br />pageNum：当type为peek时，必须大于0,当type为poll时，该参数无效<br/>pageSize：必须大于0 |
+| webhookConfig.port | Integer | 是（启用时） | 回调数据暴露端口 | |
+| webhookConfig.serverIdleTimeout | Integer | 否 | 暴露回调数据的服务器的TCP空闲超时时长 | 单位：ms，默认：5000 |
+| webhookConfig.maxStorageSize | Integer | 否 | 存储回调数据队列的最大值 | 默认：5000 |
+
+### 启动
+
+1. 启动 EventMesh Runtime
+2. 启动 eventmesh-connector-http
+
+完成之后，向 EventMesh 发送消息，然后你的消息就会通过HTTP请求发生给目标HTTP服务器了。如果启用webhook模式，则还会同时开启一个
+HTTP 服务器对外提供回调数据暴露服务。
+
+### 数据格式
+
+HTTP sink connector 发送消息的数据格式如下：
+
+- type：由connectorName、协议、模式所组成
+- time：发送请求的时间
+- uuid：请求唯一标识，在webhook模式中和回调数据结构的uuid一一对应
+- eventId：事件ID，由type和offset组成
+- data：实际需要发送的数据
+
+```json
+{
+ "type": "httpSink.http.webhook",
+ "time": "2024-05-09T22:27:02.028",
+ "uuid": "951d88ee-2503-4df3-a1dd-6e4a6a1c3d3b",
+ "eventId": "httpSink.http.webhook-0",
+ "data": ${ConnectRecord}
+}
+```
+
+启用webhook模式时，则还会同时开启一个 HTTP 服务器对外提供回调数据暴露服务。获取的回调数据格式如下：
+
+- pageNum：当前页数
+- pageSize：每页的大小
+- pageItems：获取的回调数据数组
+- pageItem的字段含义如下：
+ - data：从回调服务器返回的回调数据
+ - metadata：元数据，字段含义如下：
+ - url: 获取回调数据的URL
+ - code：请求回调数据得到的响应状态码，如果发生的网络层面的错误（无法收到响应），则为-1
+ - message：请求回调数据得到的响应信息，如果发生的网络层面的错误（无法收到响应），则为相关异常信息
+ - uuid：请求回调数据的唯一请求标识，和上面发送消息时所携带的uuid一一对应
+ - receivedTime：请求回调数据得到的响应时间
+ - retryNum：重试次数
+ - retriedBy：如果发送重试，则记录被重试的请求的uuid，否则为null
+
+```json
+{
+ "pageNum": 1,
+ "pageSize": 10,
+ "pageItems": [
+ {
+ "data": "callbackData",
+ "metadata": {
+ "url": "http://127.0.0.1:8987/test",
+ "code": 200,
+ "message": "OK",
+ "receivedTime": "2024-05-09 22:53:21.556",
+ "uuid": "fba29061-1a1f-4482-9c83-43ba4e0bcf3f",
+ "retriedBy": null,
+ "retryNum": 0
+ }
+ }
+ ]
+}
+```
+
