PIP-339: Introducing the --log-topic Option for Pulsar Sinks and Sources

Motivation

The --log-topic option already exists in Pulsar Functions, enabling users to direct function logs to a specified “log topic”. This feature is useful for debugging and analysis. However, Pulsar Sinks and Sources currently lack this option, resulting in inconsistent log management across Pulsar Functions and Connectors.

Goals

In Scope

The primary objective of this proposal is to integrate the --log-topic option into the create, update, and localrun sub-commands for Pulsar Sinks and Sources.

Detailed Design

Design & Implementation Details

  1. Integrate the --log-topic option into SinkDetailsCommand and SourceDetailsCommand:

    @Parameter(names = "--log-topic", description = "The topic to which the logs of a Pulsar Sink/Source are produced")
protected String logTopic;

  2. Pass this option to functionDetailsBuilder when creating, updating, or locally running Pulsar Sinks and Sources:

    if (sinkConfig.getLogTopic() != null) {
    functionDetailsBuilder.setLogTopic(sinkConfig.getLogTopic());
}

    if (sourceConfig.getLogTopic() != null) {
    functionDetailsBuilder.setLogTopic(sourceConfig.getLogTopic());
}

  3. Return the “log topic” when getting Pulsar Sinks and Sources

    if (!isEmpty(functionDetails.getLogTopic())) {
    sinkConfig.setLogTopic(functionDetails.getLogTopic());
}

    if (!isEmpty(functionDetails.getLogTopic())) {
    sourceConfig.setLogTopic(functionDetails.getLogTopic());
}

General Notes

Upon successful implementation of this proposal, the create, update, and localrun sub-commands for Pulsar Sinks and Sources will include the --log-topic option.

Links