[fix][consumer] Initialize batch ack tracker from the broker ack set on redelivery (#1521)

When the broker redelivers a partially-acked batch entry, it attaches the
ack set of the already-acked indexes. The client skipped delivering those
indexes but initialized the batch ackTracker as if all of them were still
outstanding. For a consumer without batch index ack, whose only way to ack
a batch is the entry-level ack gated on the tracker completing, acks of
the redelivered messages were silently dropped, so the batch was never
acked at the broker and its messages could end up in the DLQ despite
having been acked in time.

Seed the tracker from the broker-provided ack set when one is present.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
4 files changed
tree: 6909f59d80118c336d975c1ccae769296ffc5af6
  1. .github/
  2. distribution/
  3. docs/
  4. examples/
  5. integration-tests/
  6. oauth2/
  7. perf/
  8. pulsar/
  9. pulsaradmin/
  10. scripts/
  11. .asf.yaml
  12. .gitignore
  13. .golangci.yml
  14. CHANGELOG.md
  15. CONTRIBUTING.md
  16. Dockerfile
  17. go.mod
  18. go.sum
  19. LICENSE
  20. licenserc.toml
  21. Makefile
  22. NOTICE
  23. README.md
README.md

PkgGoDev Go Report Card Language LICENSE

Apache Pulsar Go Client Library

A Go client library for Apache Pulsar. For the supported Pulsar features, see Client Feature Matrix.

Purpose

This project is a pure-Go client library for Pulsar that does not depend on the C++ Pulsar library.

Once feature parity and stability are reached, this will supersede the current CGo-based library.

Requirements

  • Go 1.24+

Status

Check the Projects page at https://github.com/apache/pulsar-client-go/projects for tracking the status and the progress.

Usage

Import the client library:

import "github.com/apache/pulsar-client-go/pulsar"

Create a Producer:

client, err := pulsar.NewClient(pulsar.ClientOptions{
    URL: "pulsar://localhost:6650",
})

defer client.Close()

producer, err := client.CreateProducer(pulsar.ProducerOptions{
	Topic: "my-topic",
})

_, err = producer.Send(context.Background(), &pulsar.ProducerMessage{
	Payload: []byte("hello"),
})

defer producer.Close()

if err != nil {
    fmt.Println("Failed to publish message", err)
} else {
    fmt.Println("Published message")
}

Create a Consumer:

client, err := pulsar.NewClient(pulsar.ClientOptions{
    URL: "pulsar://localhost:6650",
})

defer client.Close()

consumer, err := client.Subscribe(pulsar.ConsumerOptions{
        Topic:            "my-topic",
        SubscriptionName: "my-sub",
        Type:             pulsar.Shared,
    })

defer consumer.Close()

msg, err := consumer.Receive(context.Background())
    if err != nil {
        log.Fatal(err)
    }

fmt.Printf("Received message msgId: %#v -- content: '%s'\n",
            msg.ID(), string(msg.Payload()))

Create a Reader:

client, err := pulsar.NewClient(pulsar.ClientOptions{URL: "pulsar://localhost:6650"})
if err != nil {
	log.Fatal(err)
}

defer client.Close()

reader, err := client.CreateReader(pulsar.ReaderOptions{
	Topic:          "topic-1",
	StartMessageID: pulsar.EarliestMessageID(),
})
if err != nil {
	log.Fatal(err)
}
defer reader.Close()

for reader.HasNext() {
	msg, err := reader.Next(context.Background())
	if err != nil {
		log.Fatal(err)
	}

	fmt.Printf("Received message msgId: %#v -- content: '%s'\n",
		msg.ID(), string(msg.Payload()))
}

Build and Test

Build the sources:

make build

Run the tests:

make test

Run the tests with specific versions of GOLANG and PULSAR:

make test GO_VERSION=1.24 PULSAR_VERSION=4.0.3

Contributing

Contributions are welcomed and greatly appreciated. See CONTRIBUTING.md for details on submitting patches and the contribution workflow.

If your contribution adds Pulsar features for Go clients, you need to update both the Pulsar docs and the Client Feature Matrix. See Contribution Guide for more details.

Community

Mailing lists
NameScope
users@pulsar.apache.orgUser-related discussionsSubscribeUnsubscribeArchives
dev@pulsar.apache.orgDevelopment-related discussionsSubscribeUnsubscribeArchives
Slack

Pulsar slack channel #dev-go at https://apache-pulsar.slack.com/

You can self-register at https://apache-pulsar.herokuapp.com/

License

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0

Troubleshooting

Go module ‘ambiguous import’ error

If you've upgraded from a previous version of this library, you may run into an ‘ambiguous import’ error when building.

github.com/apache/pulsar-client-go/oauth2: ambiguous import: found package github.com/apache/pulsar-client-go/oauth2 in multiple modules

The fix for this is to make sure you don't have any references in your go.mod file to the old oauth2 module path. So remove any lines similar to the following, and then run go mod tidy.

github.com/apache/pulsar-client-go/oauth2 v0.0.0-20220630195735-e95cf0633348 // indirect