| # PIP-259: Make the config httpMaxRequestHeaderSize of the pulsar web server to configurable |
| |
| Discuss: https://lists.apache.org/thread/f11cld5cbc8sodhgvs5s28lw8nxsr9dc |
| |
| ### Motivation |
| |
| We have two ways to manage pulsar's resources: |
| - By client API (Can manage some resources, such as `create topic`, `create subscriber`, and so on) |
| - By admin API (Can manage all the resources) |
| |
| The `client API` has no limit on the request length. And the `admin API` has a limit on the request length(such as HTTP request line and HTTP request headers), this limitation is implemented by the built-in web container Jetty. |
| |
| Almost resources can be created by two APIs, but can only be modified and deleted by `admin API`. This causes us to be unable to modify or delete resources created by `client API` with too long a name because it exceeds Jetty's default HTTP request URI length limit. |
| |
| ### Goal |
| |
| #### 1. For web servers |
| Provide a way to modify Jetty's `httpMaxRequestHeaderSize` configuration (involves two servers: the web server in pulsar and the web server in pulsar-proxy), then users can edit or delete the resources whose name is too long. |
| |
| #### 2.For the internal HTTP client in pulsar-proxy |
| Set the config `httpClientRequestBufferSize` for the internal Jetty-client's of Pulsar-proxy. |
| |
| Since the pulsar-proxy handles HTTP requests like this: `pulsar-admin.sh` -> `proxy web server` -> `(highlight) internal client in proxy` -> `pulsar web server`. |
| |
| When the internal client forwards a request, it forwards the request header and the request body, and all the data passes through a buffer( we call it Buf ), like this: |
| - Receive a request |
| - Put the request line and request headers input to the Buf. |
| - <strong>(highlight)</strong>Flush the Buf ( If the data in the request line and request header exceeds the length of the buf, an error is reported ) |
| - Put the request body input to the Buf. |
| - Flush the Buf if it is full. |
| |
| So we should set the `buff size` of the Buf: `pulsar-proxy.conf.httpClientRequestBufferSize` -> `buf size of the internal client`. |
| |
| ### API Changes |
| |
| #### ServiceConfiguration.java |
| ```java |
| @FieldContext( |
| category = CATEGORY_HTTP, |
| doc = """ |
| The maximum size in bytes of the request header. |
| Larger headers will allow for more and/or larger cookies plus larger form content encoded in a URL. |
| However, larger headers consume more memory and can make a server more vulnerable to denial of service |
| attacks. |
| """ |
| ) |
| private int httpMaxRequestHeaderSize = 8 * 1024; |
| ``` |
| |
| #### ProxyConfiguration.java |
| |
| ```java |
| @FieldContext( |
| minValue = 1, |
| category = CATEGORY_HTTP, |
| doc = """ |
| The maximum size in bytes of the request header. |
| Larger headers will allow for more and/or larger cookies plus larger form content encoded in a URL. |
| However, larger headers consume more memory and can make a server more vulnerable to denial of service |
| attacks. |
| """ |
| ) |
| private int httpMaxRequestHeaderSize = 8 * 1024; |
| ``` |
| |
| ### Anything else |
| |
| This change should cherry-pick into the previous branches ( includes `2.9~2.11` ) |
| |
| If the user uses the features `RETRY Topic` or `DLQ`, it is possible that pulsar will automatically create some topics with names that are too long and cannot be managed, the [scenario has been discussed in the email](https://lists.apache.org/thread/q1m23ckyy10wvtzy65v8bwqwnh7r0gc8) before |