Background knowledge

Apache Pulsar introduced the topic-level policies by PIP-39. It uses the reader API to read messages and relies on compaction to implement table-like logic in the memory.

PIP-104 Introduced new consumer type TableView to support the same logic by Public Stable API.

Motivation

Due to a lot of problems caused by the complex topic policy logic is as follows. we can use a new stable TableView API instead of previous one.

Goals

In Scope

  • Reduce complex logic.
  • Reuse the public stable API to improve the project quality.

Out of Scope

none.

High Level Design

sequenceDiagram
    participant client
    participant topic_policy_service
    participant system_event_topic
    client ->> topic_policy_service: Hi, what's the topic policy of `persistent://public/default/test`
    loop policy update event fetching
        topic_policy_service ->> system_event_topic: Do you have new events?;
        alt new policy event
            system_event_topic ->> topic_policy_service: Yes, Sure. here you are.
        else no policy event
            system_event_topic ->> topic_policy_service: No, Please wait for a moment.
        end;
    end
    Note right of topic_policy_service: Update the local cache table by keeping the latest one;
    topic_policy_service -->> client: Here you are.

Detailed Design

Design & Implementation Details

The new implementation will continue reuse the previous system topic namespace/__change_event to store the topic policy data. and it will change the reading logic from the raw reader to TableView to enhance the robustness.

Public-facing Changes

none.

Public API

none.

Binary protocol

none.

Configuration

none.

CLI

none.

Metrics

none.

Monitoring

none.

Security Considerations

none.

Backward & Forward Compatibility

Without backward compatibility since we will reuse the same system topic namesapce/__change_event.

Revert

none.

Upgrade

none.

Alternatives

none.

General Notes

none.

Links

none.