[FLINK-26159][doc] add description for MAX_FETCH_RECORD related question Update docs/content/docs/connectors/datastream/pulsar.md Co-authored-by: MartijnVisser <martijn@2symbols.com> Update docs/content/docs/connectors/datastream/pulsar.md Co-authored-by: MartijnVisser <martijn@2symbols.com> add Chinese documentation
diff --git a/docs/content.zh/docs/connectors/datastream/pubsub.md b/docs/content.zh/docs/connectors/datastream/pubsub.md index f766374..837b892 100644 --- a/docs/content.zh/docs/connectors/datastream/pubsub.md +++ b/docs/content.zh/docs/connectors/datastream/pubsub.md
@@ -150,4 +150,21 @@ Sink function 会把准备发到 PubSub 的信息短暂地缓存以提高性能。每次 checkpoint 前,它会刷新缓冲区,并且只有当所有信息成功发送到 PubSub 之后,checkpoint 才会成功完成。 +## 常见问题 + +由于Pulsar Connector大量依赖 +[PulsarClient](https://pulsar.apache.org/docs/en/client-libraries-java/) 和 +[PulsarAdmin](https://pulsar.apache.org/docs/en/admin-api-overview/) 来实现相关功能,有时Flink Job出现 +可能是由于Pulsar broker版本过低或者配置不合适导致的。调试配置或升级pulsar可能会解决部分问题。 + + +### 当数据量很小时,source读数据出现大约10s延迟 + +当Pulsar Source从一个数据量很小的topic读取数据时,用户可能会观测到消息间出现10秒钟的间隔。这是由于Pulsar Source +会将从broker读到的消息暂存至等待队列中,只有当队列中消息数量达到了 `PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS` +才会向下游算子发送数据。如果数据量一直很小,那么就会在等待直到等待时长超过`PULSAR_MAX_FETCH_TIME`规定的值 (默认10秒钟)时 +也向下游算子发送数据。 + +为了避免这种情况,用户需要改变两个配置项的值。 + {{< top >}}
diff --git a/docs/content/docs/connectors/datastream/pulsar.md b/docs/content/docs/connectors/datastream/pulsar.md index 680fb1a..6e89174 100644 --- a/docs/content/docs/connectors/datastream/pulsar.md +++ b/docs/content/docs/connectors/datastream/pulsar.md
@@ -417,4 +417,13 @@ and your problem might be independent of Flink and sometimes can be solved by upgrading Pulsar brokers, reconfiguring Pulsar brokers or reconfiguring Pulsar connector in Flink. +### Messages can be delayed on low volume topics + +When the Pulsar source connector reads from a low volume topic, users might observe a 10 seconds delay between messages. Pulsar buffers messages from topics by default. Before emitting to downstream +operators, the number of buffered records must be equal or larger than `PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS`. If the data volume is low, it could be that filling up the number of buffered records takes longer than `PULSAR_MAX_FETCH_TIME` (default to 10 seconds). If that's the case, it means that only after this time has passed the messages will be emitted. + +To avoid this behaviour, you need to change either the buffered records or the waiting time. + + + {{< top >}}