| # PIP-131: Resolve produce chunk messages failed when topic level maxMessageSize is set |
| |
| Discussion thread: https://lists.apache.org/thread/pp1lgpbhy07z5mg2hrosrbjwksxx3qq8 |
| Vote thread: https://lists.apache.org/thread/cgww0m8dkrb14dbmkmw2qq3v5kkgvdmc |
| |
| ## Motivation |
| |
| Currently, chunk messages producing fails if topic level maxMessageSize is set [1]. The root cause of this issue is because chunk message is using broker level maxMessageSize as chunk size. And topic level maxMessageSize is always <= broker level maxMessageSize. So once it is set, the on-going chunk message producing fails. |
| |
| ## Goal |
| |
| Resolve topic level maxMessageSize compatibility issue with chunking messages. |
| |
| ## Implementation |
| |
| Current best solution would be just skipping topic level maxMessageSize check in org.apache.pulsar.broker.service.AbstractTopic#isExceedMaximumMessageSize. |
| Topic level maxMessageSize is introduced in [2], for the purpose of "easier to plan resource quotas for client allocation". And IMO this change will not bring further complex into this. |
| |
| ## Reject Alternatives |
| |
| Add a client side topic level maxMessageSize and keep it synced with broker. |
| |
| Required changes: |
| - [client] Add a new field org.apache.pulsar.client.impl.ProducerBase#maxMessageSize to store this client side topic level maxMessageSize. |
| - [PulsarApi.proto] Add a new field maxMessageSize in the CommandProducerSuccess for the initial value of ProducerBase#maxMessageSize |
| - [PulsarApi.proto] Add a new Command like CommandUpdateClientPolicy{producerId, maxMessageSize} to update ProducerBase#maxMessageSize when topic level maxMessageSize is updated. |
| Further more, some other data consistency issues need be handled very carefully when maxMessageSize is updated. |
| This alternative is complex but can also solve other topic level maxMessageSize issue [3] when batching is enabled (non-batching case is solved with PR [4]). |
| |
| [1] https://github.com/apache/pulsar/issues/13360 |
| [2] https://github.com/apache/pulsar/pull/8732 |
| [3] https://github.com/apache/pulsar/issues/12958 |
| [4] https://github.com/apache/pulsar/pull/13147 |
