Google Git
Sign in
apache / pulsar / 7c96a36c58768e71cc445371bb3d98c5ac6e05cd
commit7c96a36c58768e71cc445371bb3d98c5ac6e05cd[log] [tgz]
authorfengyubiao <yubiao.feng@streamnative.io>Mon Aug 21 04:49:30 2023 +0800
committerGitHub <noreply@github.com>Mon Aug 21 04:49:30 2023 +0800
treebbcd1a06c9fafe010884fb84629d648183368640
parent00ca6aa1cef950bd9ea5014d4332d16d3c8d37e2 [diff]
[fix] [admin] Fix get topic stats fail if a subscription catch up concurrently (#20971)

### Motivation

**Background**: when calling `pulsar-admin topics stats --get-earliest-time-in-backlog <topic name>`, Pulsar will read the first entry which is not acknowledged, and respond with the entry write time. The flow is like this:
- get the mark deleted position of the subscription
- if no backlog, response `-1`
- else read the next position of the mark deleted position, and respond with the entry write time.

**Issue**: if the command `pulsar-admin topics stats --get-earliest-time-in-backlog <topic name>` and `consumer.acknowledge` are executed at the same time, the step 2 in above flow will get a position which is larger than the last confirmed position, lead a read entry error.

| time | `pulsar-admin topics stats --get-earliest-time-in-backlog <topic name>` | `consumer.acknowledge` |
| --- | --- | --- |
| 1 | mark deleted position is `3:1` and LAC is `3:2` now |
| 2 | the check `whether has backlog` is passed |
| 3 | | acknowledged `3:2`, mark deleted position is `3:2` now |
| 4 | calculate next position: `3:3` |
| 5 | Read `3:3` and get an error: `read entry failed` |

Note: the test in PR is not intended to reproduce the issue.

### Modifications

Respond `-1` if the next position of the mark deleted position is larger than the LAC
4 files changed
tree: bbcd1a06c9fafe010884fb84629d648183368640
  1. .github/
  2. .idea/
  3. .mvn/
  4. bin/
  5. bouncy-castle/
  6. build/
  7. buildtools/
  8. conf/
  9. deployment/
  10. dev/
  11. distribution/
  12. docker/
  13. docker-compose/
  14. grafana/
  15. jclouds-shaded/
  16. managed-ledger/
  17. pip/
  18. pulsar-broker/
  19. pulsar-broker-auth-athenz/
  20. pulsar-broker-auth-oidc/
  21. pulsar-broker-auth-sasl/
  22. pulsar-broker-common/
  23. pulsar-client/
  24. pulsar-client-1x-base/
  25. pulsar-client-admin/
  26. pulsar-client-admin-api/
  27. pulsar-client-admin-shaded/
  28. pulsar-client-all/
  29. pulsar-client-api/
  30. pulsar-client-auth-athenz/
  31. pulsar-client-auth-sasl/
  32. pulsar-client-cpp/
  33. pulsar-client-messagecrypto-bc/
  34. pulsar-client-shaded/
  35. pulsar-client-tools/
  36. pulsar-client-tools-api/
  37. pulsar-client-tools-customcommand-example/
  38. pulsar-client-tools-test/
  39. pulsar-common/
  40. pulsar-config-validation/
  41. pulsar-docs-tools/
  42. pulsar-function-go/
  43. pulsar-functions/
  44. pulsar-io/
  45. pulsar-metadata/
  46. pulsar-package-management/
  47. pulsar-proxy/
  48. pulsar-sql/
  49. pulsar-testclient/
  50. pulsar-transaction/
  51. pulsar-websocket/
  52. src/
  53. structured-event-log/
  54. testmocks/
  55. tests/
  56. tiered-storage/
  57. wireshark/
  58. .asf.yaml
  59. .gitignore
  60. codecov.yml
  61. CONTRIBUTING.md
  62. CONTRIBUTORS.md
  63. faq.md
  64. LICENSE
  65. lombok.config
  66. mvnw
  67. mvnw.cmd
  68. NOTICE
  69. pom.xml
  70. README.md
  71. SECURITY.md
README.md

logo

docker pull contributors last commit release downloads

Pulsar is a distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API.

Learn more about Pulsar at https://pulsar.apache.org

Main features

  • Horizontally scalable (Millions of independent topics and millions of messages published per second)
  • Strong ordering and consistency guarantees
  • Low latency durable storage
  • Topic and queue semantics
  • Load balancer
  • Designed for being deployed as a hosted service:
    • Multi-tenant
    • Authentication
    • Authorization
    • Quotas
    • Support mixing very different workloads
    • Optional hardware isolation
  • Keeps track of consumer cursor position
  • REST API for provisioning, admin and stats
  • Geo replication
  • Transparent handling of partitioned topics
  • Transparent batching of messages

Repositories

This repository is the main repository of Apache Pulsar. Pulsar PMC also maintains other repositories for components in the Pulsar ecosystem, including connectors, adapters, and other language clients.

Helm Chart

Ecosystem

Clients

Dashboard & Management Tools

Website

CI/CD

Archived/Halted

Pulsar Runtime Java Version Recommendation

  • pulsar ver > 2.10 and master branch
ComponentsJava Version
Broker17
Functions / IO17
CLI17
Java Client8 or 11 or 17
  • 2.8 <= pulsar ver <= 2.10
ComponentsJava Version
Broker11
Functions / IO11
CLI8 or 11
Java Client8 or 11
  • pulsar ver < 2.8
ComponentsJava Version
All8 or 11

Build Pulsar

Requirements

  • JDK

    Pulsar VersionJDK Version
    master and 2.11 +JDK 17
    2.8 / 2.9 / 2.10JDK 11
    2.7 -JDK 8

  • Maven 3.6.1+

  • zip

Note:

This project includes a Maven Wrapper that can be used instead of a system-installed Maven. Use it by replacing mvn by ./mvnw on Linux and mvnw.cmd on Windows in the commands below.

It's better to use CMD rather than Powershell on Windows. Because maven will activate the windows profile which runs rename-netty-native-libs.cmd.

Build

Compile and install:

$ mvn install -DskipTests

Compile and install individual module

$ mvn -pl module-name (e.g: pulsar-broker) install -DskipTests

Minimal build (This skips most of external connectors and tiered storage handlers)

mvn install -Pcore-modules,-main -DskipTests

Run Unit Tests:

$ mvn test

Run Individual Unit Test:

$ mvn -pl module-name (e.g: pulsar-client) test -Dtest=unit-test-name (e.g: ConsumerBuilderImplTest)

Run Selected Test packages:

$ mvn test -pl module-name (for example, pulsar-broker) -Dinclude=org/apache/pulsar/**/*.java

Start standalone Pulsar service:

$ bin/pulsar standalone

Check https://pulsar.apache.org for documentation and examples.

Build custom docker images

  • Docker images must be built with Java 8 for branch-2.7 or previous branches because of ISSUE-8445.
  • Java 11 is the recommended JDK version in branch-2.8, branch-2.9 and branch-2.10.
  • Java 17 is the recommended JDK version in master.

The following command builds the docker images apachepulsar/pulsar-all:latest and apachepulsar/pulsar:latest:

mvn clean install -DskipTests
mvn package -Pdocker,-main -am -pl docker/pulsar-all -DskipTests

After the images are built, they can be tagged and pushed to your custom repository. Here's an example of a bash script that tags the docker images with the current version and git revision and pushes them to localhost:32000/apachepulsar.

image_repo_and_project=localhost:32000/apachepulsar
pulsar_version=$(mvn initialize help:evaluate -Dexpression=project.version -pl . -q -DforceStdout)
gitrev=$(git rev-parse HEAD | colrm 10)
tag="${pulsar_version}-${gitrev}"
echo "Using tag $tag"
docker tag apachepulsar/pulsar-all:latest ${image_repo_and_project}/pulsar-all:$tag
docker push ${image_repo_and_project}/pulsar-all:$tag
docker tag apachepulsar/pulsar:latest ${image_repo_and_project}/pulsar:$tag
docker push ${image_repo_and_project}/pulsar:$tag

Setting up your IDE

Read https://pulsar.apache.org/contribute/setup-ide for setting up IntelliJ IDEA or Eclipse for developing Pulsar.

Documentation

Note:

For how to make contributions to Pulsar documentation, see Pulsar Documentation Contribution Guide.

Contact

Mailing lists
NameScopeSubscribeUnsubscribeArchives
users@pulsar.apache.orgUser-related discussionsSubscribeUnsubscribeArchives
dev@pulsar.apache.orgDevelopment-related discussionsSubscribeUnsubscribeArchives
Slack

Pulsar slack channel at https://apache-pulsar.slack.com/

You can self-register at https://communityinviter.com/apps/apache-pulsar/apache-pulsar

Report a security vulnerability

To report a vulnerability for Pulsar, contact the Apache Security Team. When reporting a vulnerability to security@apache.org, you can copy your email to private@pulsar.apache.org to send your report to the Apache Pulsar Project Management Committee. This is a private mailing list.

https://github.com/apache/pulsar/security/policy contains more details.

License

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0

Crypto Notice

This distribution includes cryptographic software. The country in which you currently reside may have restrictions on the import, possession, use, and/or re-export to another country, of encryption software. BEFORE using any encryption software, please check your country's laws, regulations and policies concerning the import, possession, or use, and re-export of encryption software, to see if this is permitted. See The Wassenaar Arrangement for more information.

The U.S. Government Department of Commerce, Bureau of Industry and Security (BIS), has classified this software as Export Commodity Control Number (ECCN) 5D002.C.1, which includes information security software using or performing cryptographic functions with asymmetric algorithms. The form and manner of this Apache Software Foundation distribution makes it eligible for export under the License Exception ENC Technology Software Unrestricted (TSU) exception (see the BIS Export Administration Regulations, Section 740.13) for both object code and source code.

The following provides more details on the included cryptographic software: Pulsar uses the SSL library from Bouncy Castle written by http://www.bouncycastle.org.