Apache
2.3.2 — 2019-05-30
Fixes
- Validate admin operation on topic with authoritative parameter #4270
- fix bug with source local run #4278
- fix cannot use size (type _Ctype_int) as type _Ctype_ulong #4212
- Fix segfault in c++ producer #4219
- AlwaysCompatible doesn't use AlwaysSchemaValidator in 2.3.1 component/schemaregistry #4181
- Avoid potentially blocking method during topic ownership check #4190
- [pulsar-broker]Fix: client-producer can't reconnect due to failed producer-future on cnx cache #4138
- Removing # TYPE comment from topic metrics in Prometheus #4136
- For functions metrics in prometheus also remove TYPE #4081
- Fix: set receive queue size for sinks #4091
- Fix: Exception when switch cluster from auth enabled to auth disabled #4069
- Fix update cli source sink #4061
- Fix connectors nested configs #4067
- For functions metrics, avoid having HELP #4029
- Fix Python functions state which is completely broken #4027
- [issue #3975] Bugfix NPE on non durable consumer #3988
- Fix: Function auth should ignore exception because it might be anonymous user #4185
- [pulsar-function] fix worker-stats broken admin-api #4068
- fix errors in sql doc #4030
- Fix the swagger files generated by removing troublesome class #4024
- [pulsar-function] fix broken backward compatibility with v1-namespace while registering function #4224
- Revert dup consumer and related code #4142
- [issue 4274][pulsar-io]Add double quotation marks for metrics with remote_cluster #4295
Enhancements
- By default, auto configure the size of Bookie read/write cache #4297
- Upgrade to BookKeeper 4.9.2 #4288
- [pulsar-function] support bookie authentication from function-worker #4088
- Optimizing performance for Pulsar function archive download #4082
- allow users to update output topics for functions and sources #4092
- improve data-generator source performance #4058
- [client] Set actual topic name to partitioned consumer #4064
- ack records in datagenerator print sink #4052
- [security] Upgrade athenz libraries #4056
- [python client] Handle subrecords in JsonSchema encoding #4023
- [Issue 3987][pulsar-broker]Handle config is null when create tenant #4019
- Add bookkeeper client version constraint #4013
- Improve error handling for triggering function when there is a schema mismatch #3995
- [pulsar-broker] add producer/consumer id in error-logging #3961
2.3.1 — 2019-04-12
Fixes
- Fixed C++ batch acks tracker to evict message from sendList array. This was causing a slowdown in C++ consumers #3618
- Allow publishing messages >5MB with batching (when they compress to <5MB) #3673 and #3718
- Use at least 8 threads in Jetty thread pool. This fixed deadlocks in Jetty requests handling #3776
- Fixed Reader.HasNext() in Go client #3764
- Fixed increasing consumer permits after ack dedup operation. #3787
- Set the dedup cursor as “inactive” after recovery #3612
- Fix read batching message by pulsar reader #3830
- Fix submit function with code specified via URL #3934
- Fixed reader reading from a partition #3960
- Fixed issue with Authorization header missing after client gets redirected #3869
Enhancements
- Added
producer.flush()
on Python #3685 - Introduced schema builder to define schema #3682
- Support passing schema definition for JSON and AVRO schemas #3766
- Exposing InitialPosition setting in Python consumer #3714
For a complete list of issues fixed, see
https://github.com/apache/pulsar/milestone/21?closed=1
https://github.com/apache/pulsar/releases/tag/v2.3.1
2.3.0 — 2019-02-20
General
- Support for schema definitions in the Pulsar Python client library
- PIP-25: Token based authentication #2888, #3067 and #3089
- Updated to Apache BookKeeper 4.9.0
- ZStandard compression codec #3159. Note that when a producer choose Zstd compression, a consumer will need to be at least at version 2.3.0 to be able to correctly receive the messages.
- Support for Java 11 #3006
- Added
Schema.AUTO_PRODUCE
type to allow to publish serialized data and validate it against the topic schema #2685 - Added
Schema.KeyValue
to allow for schema to be validated on message keys as well as payloads. #2885 - Support TLS authentication and authorization in standalone mode #3360
- When creating namespace, use local cluster by default #3571
- Tag BookKeeper ledgers created by Pulsar with topic/subscription names for info/debug purposes #3525
- Enabled sticky reads in BooKeeper reads to increase IO efficiency with read-ahead #3569
- Several optimization in Pulsar SQL Presto connector (#3128, #3135, #3139, #3144, #3143)
- Configure Pulsar broker data cache automatically from JVM settings #3573
- Reuse the SSL context objects #3550
- Automatic schema update can be disabled through admin interface #2691
- Support Dead-Letter-Queue from WebSocket proxy #2968
- Pull-mode for WebSocket proxy #3058
- Export Jetty stats to Prometheus #2804
- Added stats for Pulsar proxy #2740
- Allow subscribers to access subscription admin-api #2981
- Make brokers read on closest Bookie in a multi-region deployment #3171
Fixes
- Fixed deadlock in reusing ZookKeeper event thread #3591
- In functions log topic appender, don't set producer name #3544
- When cursor recovery encounters empty cursor ledger, fallback to latest snapshot #3487
- Fixed C++ regex-consumer when using HTTP service URL #3407
- Fix race condition: broker not scheduling read for active consumer #3411
Pulsar IO
- Added Debezium connector for Change-Data-Capture into Pulsar #2791
- Added MongoDB connector #3561
- Added Elastic Search connector #2546
- Added HBase sink #3368
- Added Local files connector #2869
- Report source/sink stats in Prometheus #3261
- Allow filtering in Twitter Firehose connector #3298
- Sources/Sinks can be launched using fat jars as well #3166
Pulsar Functions
- Added Kubernetes runtime #1950
- Secrets interface #2826
- Cleanup subscriptions when deleting functions #3299
- Add Windowfunction interface to functions api #3324
- Support for accessing state in Python #2714
- Support submitting Python functions as wheel file
- Support submitting Python functions as Zip file with dependencies included #3321
- Add minimum amount of resources to run setting for functions #3536
- Fixed the behavior of Function start/stop #3477
Java client
Python client
- Fixed
Message.properties()
#3595
Go client
- Added
Producer.flush()
to flush all outstanding messages #3469 - Support
Consumer.Seek()
#3478 - Added
Message.Topic()
#3346 - Allow to specify
SubscriptionInitPos
option in ConsumerOptions
#3588 - Added TLS hostname verification #3580
- Allow to link statically against
libpulsar.a
#3488 - Expose
Producer.LastSequenceID()
and Message.SequenceID()
#3416
C++ client
- Enable batching by default when using
sendAsync()
#2949 - Allow to specify schema info in Avro format #3354
- Added
Producer.flush()
to flush all outstanding messages #3020 - Added TLS hostname verification #2475
- Allow to specify
SubscriptionInitialPosition
#3567 - Added
Message.getTopicName()
#3326 - Added
Cosnsumer.receiveAsync()
#3389 - Build
libpulsar.a
with all required dependencies #3488 - Removed Boost from Pulsar API headers #3374
Adaptors
For a complete list of issues fixed, see
https://github.com/apache/pulsar/milestone/18?closed=1
https://github.com/apache/pulsar/releases/tag/v2.3.0
2.2.1 — 2018-12-31
This release includes fixes for 2.2.0 release. In particular:
Fixed issue when proxy HTTP admin API requests Pulsar proxy #3022
Fixed Consumer.unsubscribe()
in Python client library #3093
Fixed ZLib decompression in C++ client #2854
Fixed Pulsar functions context publish in Python #2844
For a complete list of issues fixed, see
https://github.com/apache/pulsar/milestone/19?closed=1
https://github.com/apache/incubator-pulsar/releases/tag/v2.2.1
2.2.0 — 2018-10-24
This is the first release of Pulsar as an Apache Top Level Project
This is a feature release, including several new features, improvements and fixes for issues reported for 2.1.1-incubating.
For a complete list of issues fixed, see
https://github.com/apache/pulsar/milestone/16?closed=1
https://github.com/apache/incubator-pulsar/releases/tag/v2.2.0
Apache incubator
2.1.1-incubating — 2018-09-17
This release fixes issues reported for 2.1.0-incubating.
- #2473 - Downgrading ZK to stable version 3.4.13
- #2219 - Cpp client: add PatternMultiTopicsConsumerImpl to support regex subscribe
- #2387 - Fixed race condition during expansion of concurrent open hash maps
- #2348 - Fix NPE when splitting and unloading bundle
- #2223 - fix bug in FunctionRuntimeManager involving not cleaning up old invalid assignments
- #2367 - [compaction] make topic compaction works with partitioned topic
- #2203 - Make sure schema is initialized before the topic is loaded
The complete list of changes can be found at: https://github.com/apache/incubator-pulsar/milestone/17?closed=1
https://github.com/apache/incubator-pulsar/releases/tag/v2.1.1-incubating
2.1.0-incubating — 2018-08-02
This is the seventh release of Apache Pulsar since entering the ASF incubator.
It is a feature release, including several new features and major improvements:
The complete list of changes can be found at: https://github.com/apache/incubator-pulsar/milestone/13?closed=1
https://github.com/apache/incubator-pulsar/releases/tag/v2.1.0-incubating
2.0.1-incubating — 2018-06-18
This release fixes issues reported for 2.0.0-rc1-incubating.
- #1893 - Fixed issues with Python packages on PyPI
- #1797 - Proxy doesn't strip the request URL for admin requests correctly
- #1862 - Fix REST APIs provided by Pulsar proxy
The complete list of changes can be found at: https://github.com/apache/incubator-pulsar/milestone/14?closed=1
https://github.com/apache/incubator-pulsar/releases/tag/v2.0.1-incubating
1.22.1-incubating — 2018-06-18
This is the sixth release of Apache Pulsar since entering the ASF incubator.
This release addresses issues reported in 1.22.0-incubating version.
- #1660 Deadlock while closing non persistent topic
- #1591 Deadlock while closing non shared consumer
- #1554 Handle invalid mark delete position at managed cursor
- #1262 Broker should not start replicator for root partitioned topic
- #1662 NPE when cursor failed to close empty subscription
- #1370 Relocate service files for shading pulsar-client-admin module
- #1265 Fixed lookup redirect logic on Proxyside
- #1428 Handle Race condition in concurrent bundle split
- #1817 Fixed mem leak when acknowledging while disconnected from broke
- #1851 Fixing resource leak due to open file descriptors in SecurityUtility.
The complete list of changes can be found at: https://github.com/apache/incubator-pulsar/milestone/15?closed=1
https://github.com/apache/incubator-pulsar/releases/tag/v1.22.1-incubating
2.0.0-rc1-incubating — 2018-05-29
This is the fifth release of Apache Pulsar since entering the ASF incubator and the first time we increase major release number.
There are several new features and major improvements:
- Pulsar functions: Lightweight compute framework
- New type-safe Java API for producer/consumers
- Schema registry — Enforce schema on topics
- Topic compaction — Out of band compaction of messages to allow consumer to fetch a snapshot with last published message for each message key.
- Upgraded to Apache BookKeeper 4.7.0
- Performance improvements — Up to 3x throughput improvements compared to Pulsar-1.22 and 99.9 Pct publish latencies <10ms
- Simplified terminology and admin tools
- Renamed “property” into “tenants”
- Short topic names:
my-topic
- Topics independent of cluster names:
my-tenant/my-namespace/my-topic
The complete list of changes can be found at: https://github.com/apache/incubator-pulsar/milestone/12?closed=1
https://github.com/apache/incubator-pulsar/releases/tag/v2.0.0-rc1-incubating
1.22.0-incubating — 2018-03-06
This is the fourth of Apache Pulsar since entering the ASF incubator.
Major changes in this release include:
Features
- #896 PIP-7 Introduce Failure-domain and Anti-affinity-namespace group
- #1031 Add optional key/value metadata to producers/consumers
- #1129 Added end to end encryption in C++ client
- #1151 Added REST handler to create a subscription on a topic
- #1087 Add basic authentication plugin
- #1200 Add pluggable authorization mechanism
- #1208 Add hostname-verification at client tls connection
- #950 Provided an DCOS Universe package for pulsar
- #1046 Introduce config to skip non-recoverable data-ledger
- #899 Add subscription auth mode by prefix
- #1135 Added infinite time retention configuration option
Enhancements
- #1094 Include BoringSSL native implementation for faster TLS
- #1204 Reduce size of buffer used to assemble batches
- #930 Perform async DNS resolution
- #1124 Support Pulsar proxy from C++/Python client library
- #1012 Made load shedding for load manager Dynamically configurable
- #962 Raw Reader for Pulsar Topics
- #941 Upgraded Jackson version
- #1002, #1169, #1168 Making Pulsar Proxy more secure
- #1029 Fix MessageRouter hash inconsistent on C++/Java client
Fixes
- #1153 Fixed increase partitions on a partitioned topic
- #1195 Ensure the checksum is not stripped after validation in the broker
- #1203 Use duplicates when writing from ByteBuf pair to avoid multiple threads issues
- #1210 Cancel keep-alive timer task after the proxy switch to TCP proxy
- #1170 Upgrade BK version: BK-4.3.1.91-yahoo (fix: stats + DoubleByteBuf)
- #875 Bug fixes for Websocket proxy
The complete list of changes can be found at: https://github.com/apache/incubator-pulsar/milestone/11?closed=1
https://github.com/apache/incubator-pulsar/releases/tag/v1.22.0-incubating
1.21.0-incubating — 2017-12-17
This is the third of Apache Pulsar since entering the ASF incubator.
Major changes in this release include:
- #689 Upgrade to Netty 4.1
- #846 Publish the shaded pulsar-client as the default dependency
- #832 #833 #849 #852 Enhancements to Kafka API wrapper to have it work with Kafka's own benchmark tools
- #836 Fix to C++ partitioned consumer client
- #822 #826 Several fixes and improvements related to the namespace bundles
- #848 Allow consumer to seek to message id from within Pulsar client
- #903 PIP-8: Scale Pulsar beyond 1M topics
- #824 Enable secure replication over TLS
- #923 Upgrade to bk-4.3.1.83-yahoo to expose journalSyncData option
- #807 Prevent message duplication when active consumer is changed
Complete list of changes can be found at: https://github.com/apache/incubator-pulsar/milestone/10?closed=1
https://github.com/apache/incubator-pulsar/releases/tag/v1.21.0-incubating
1.20.0-incubating — 2017-08-08
This is the second of Apache Pulsar since entering the ASF incubator.
Major changes in this release include:
Complete list of changes can be found at: https://github.com/apache/incubator-pulsar/milestone/9?closed=1
https://github.com/apache/incubator-pulsar/releases/tag/v1.20.0-incubating
1.19.0-incubating — 2017-08-08
This is the first of Apache Pulsar since entering the ASF incubator.
Major changes included in this release are:
- #524 Moved APIs from
com.yahoo.pulsar
to org.apache.pulsar
- #548 Added stateless Pulsar proxy
- #538 Support for non-persistent topics
- #587 Upgraded RocksDB to comply with ASF policy
- #507 Instrumentation of ZooKeeper client to expose metrics
- Various fixes for TLS auth in WebSocket proxy
Complete list of changes can be found at: https://github.com/apache/incubator-pulsar/milestone/8?closed=1
https://github.com/apache/incubator-pulsar/releases/tag/v1.19.0-incubating
Pre-Apache
1.18 — 2017-06-17
Main changes:
- #325 Add Modular load manager documentation
- #329 Add api to get list of partitioned topics
- #296 Added spark streaming custom receiver for pulsar
- #317 HTTP lookups for c++ client lib
- #332 Fix: Modular load manager bug fixes
- #352 Fix: Delete local-policies and invalidate cache when namespace is dele
- #356 Fix: WebSocket TLS connection bug
- #363 Use binary protocol lookup for connection between WebSocket proxy and broker
- #375 Fix: Bug fixes on deadlock while topic loading failure
- #376 Fix: Avoid incrementing unack-msg count for non-shared sub and not show it on stats
- #329 Fix: Handle zkCache failures
- #387 Pass client library version to broker and show on stats
- #345 Add load shedding strategy
- #393 Change default mark-delete rate limit from 10s to 1s
- #392 Upgrade to netty-4.0.46
- #366 NonDurable cursor for managed ledger
- #371 Introduce topic reader in client API
- #341 Add stats and monitoring for websocket proxy
- #299 Add api to increase partitions of existing non-global partitioned-topic
- #294 Add endpoint to fetch stats for Prometheus
- #440 Enable PulsarAdmin to trust multiple certificates
- #442 Fix: Remove broker weights for ModularLoadManager
- #446 Fix: Recover cursor with correct readPosition and replay unackedMessages
- #441 Set Block If queue full to false by default
- #447 Fix: DoubleByteBuf to send large size messages in TLS mode
- #443 Add topic termination option
- #436 Added ZooKeeper instrumentation for enhanced stats
- #448 WebSocket proxy should not make a consumer/producer when authorization is failed
- #443 Add Docker images definition and instruction to deploy on Kubernetes
- #474 Fix: message rate out with batches to count messages/s
- #482 Allow client(producer/consumer) to check topic stats
- #468 Pulsar Python client library
- #386 Increment bookkeeper version to 4.3.1.69-yahoo
Full list of changes: https://github.com/yahoo/pulsar/milestone/7?closed=1
https://github.com/apache/incubator-pulsar/releases/tag/v1.18
1.17.5 — 2017-05-02
- #343 Fix ModularLoadManager to select broker from current available-broker list
- #384 Fix Send replay entries read callback from background thread, to avoid recursive stack calls
- #390 Fix Shaded AsyncHttpClient in pulsar client
- #374 Fix Remove Exceptionally Completed Topic Futures
https://github.com/apache/incubator-pulsar/releases/tag/v1.17.5
1.17.4 — 2017-04-25
- #362 Fix add timeout on blocking ZookeeperCache get call
- #375 Fix possible deal lock on topic loading if broker fails to get MLConfiguration from zk
- #377 Fix zkCache error handling and zk-callback processing on separate dedicated thread
https://github.com/apache/incubator-pulsar/releases/tag/v1.17.4
1.17.3 — 2017-04-20
- #367 Fix dispatcher correctly finds available consumer from list of shared-subscription consumers
https://github.com/apache/incubator-pulsar/releases/tag/v1.17.3
1.17.2 — 2017-04-06
- #327 Create znode for dynamic configuration if not present
- #336 Fix prevent creation of topic when bundle is disable
- #338 Fix deserialize load report based on load-manager
https://github.com/apache/incubator-pulsar/releases/tag/v1.17.2
1.17.1 — 2017-03-30
- #326 Fix memory leak while duplicating entry data from existing entry
https://github.com/apache/incubator-pulsar/releases/tag/v1.17.1
1.17 — 2017-03-30
Main changes:
- #188 Pulsar Dashboard
- #276 Broker persist individually deleted messages
- #282 Support binary format to persist managed-ledger info in ZK
- #292 Added REST and CLI tool to expose ManagedLedger metadata
- #285 Add documentation in japanese
- #178 Add Athenz authentication plugin
- #186 Update Broker service configuration dynamically
- #215 Fix Broker disconnects unsupported batch-consumer on batch-message topic
- #165 Message dispatching on consumer priority-level
- #303 Introduce new load manager implementation
- #306 Add topic loading throttling at broker
- #302 Update BK version to 4.3.1.60-yahoo to include: 64bit ledger-ids, fix: memory leak on read-only bookie and datasketches concurrency issue
- #216 Binary proto api to get consumer stats
- #225 Server lookup throttling
- #182 Client lookup request throttling and server-error handling
- #265 Fix client handling on http server error
- #204 Fix discovery service redirection
- #311 Fix netty package conflict at binary distribution
- #221 Fixed race condition on client reconnection logic
- #239 Fix replicator handling on closed cursor
- #318 GC improvements: Recyclable entry and reduce collection on stats generation
Full list of changes: https://github.com/apache/incubator-pulsar/milestone/3?closed=1
https://github.com/apache/incubator-pulsar/releases/tag/v1.17
1.16.5 — 2017-03-10
- #311 Exclude netty individual jars from binary distribution. This issue was causing binary distribution to have conflicting netty dependencies.
https://github.com/apache/incubator-pulsar/releases/tag/v1.16.5
1.16.4 — 2017-03-10
- #265 Fix client closes http-connection on internal-server error
- #283 Fix recycle keep alive command-object properly
- #284 Reduce usage of collections in managed-ledger metrics-generation to reduce GC impact
https://github.com/apache/incubator-pulsar/releases/tag/v1.16.4
1.16.3 — 2017-03-01
- #275 Fix for Infinite loop in PersistentReplicator.startProducer()
https://github.com/apache/incubator-pulsar/releases/tag/v1.16.3
1.16.2 — 2017-02-24
- #250 : Disconnect consumers without closing dispatcher on cursor-reset
https://github.com/apache/incubator-pulsar/releases/tag/v1.16.2
1.16.1 — 2017-02-24
- #221 Fixed race condition while creating client connection
- #223 Fixed broker's direct memory usage count
- #220 Fixed stuck replicator producer on backlog quota exception
- #239 Fixed replicator stop reading on already closed cursor
https://github.com/apache/incubator-pulsar/releases/tag/v1.16.1
1.16 — 2017-02-02
Main changes:
- #76 Async Zookeeper cache implementation
- #105 Support topic lookup using pulsar binary protocol
- #164 Fixed handling failure of unloading namespace bundle
- #166 Support websocket proxy deployment without passing globalZK
- #161 Fixed avoiding creation of duplicate replicator
- #160 Add support uri encoding on broker admin rest api
- #143 Include DataSketches metrics provider for bookie stats
- #127 Updated BK-4.3.1.45/47-yahoo to include bookie/bookkeeper-client bug-fixes and DataSketch metrics provider
- #124 Consumer-stats: Add blockedConsumer flag
- #95 Consumer-stats: Add message redelivery rate
- #123 Fixed Batch message replication
- #106 Fixed Partitioned consumer should avoid blocking call to fill shared queue
- #139 Support online consumer cursor reset
- #187 Support custom advertised address in pulsar standalone
Full list of changes: https://github.com/yahoo/pulsar/milestone/2?closed=1
https://github.com/apache/incubator-pulsar/releases/tag/v1.16
1.15.7 — 2017-01-25
- #174 Handling bundle unloading failure
https://github.com/apache/incubator-pulsar/releases/tag/v1.15.7
1.15.6 — 2017-01-20
- #171 Fix: Consumer redelivery should not wipeout availablePermits
https://github.com/apache/incubator-pulsar/releases/tag/v1.15.6
1.15.5 — 2017-01-03
- #159 Fix: Replicator-cleanup while closing replicator at broker.
- #160 Fix: Http lookup for topic with special character
https://github.com/apache/incubator-pulsar/releases/tag/v1.15.5
1.15.4 — 2016-12-14
- #146 Fix: Partitioned consumer can consume messages with receiverQueueSize 1.
https://github.com/apache/incubator-pulsar/releases/tag/v1.15.4
1.15.3 — 2016-12-13
- #145 Fixed issue Partitioned-consumer aggregate messages without blocking internal listener thread
https://github.com/apache/incubator-pulsar/releases/tag/v1.15.3
1.15.2 — 2016-11-03
- #102 Fixed issue with message dispatching while message-replay at broker
https://github.com/apache/incubator-pulsar/releases/tag/v1.15.2
1.15.1 — 2016-10-27
- #89 Fixed issue with replication in a mixed environment with 1.14 and 1.15 brokers
https://github.com/apache/incubator-pulsar/releases/tag/v1.15.1
1.15 — 2016-10-18
- #39 Updated BookKeeper version to 4.3.1.41-yahoo to include bookie storage improvements
- #17 Fixed memory leak in stats generation buffer rollover
- #27 Fixed issues with discovery service component when HTTPS is enabled
- #43 Add end-to-end crc32c checksum verification on message header and payload, rather than just payload. Support for intel hardware instructions to speed up computation.
- #26 Added ability to configure the address that the broker uses to advertise itself. Needed in cases where the public hostname/ip is different than the machine interface ip (eg: in AWS EC2 instances).
- #38 Prevent message-replay of already acknowledged messages
- #51 Per message unacknowledged redelivery. When ack-timeout is configured, only request redelivery of messages that effectively have the timeout expired, instead of all the messages dispatched to the consumer.
- #48 Add unacknowledged messages threshold to stop delivery to consumer that are not acknowledging messages
- #59 Added admin method to do a one-time messages time expiration for a given subscription (independently from the TTL configured at the namespace level)
Full list of changes: https://github.com/apache/incubator-pulsar/milestone/1?closed=1
https://github.com/apache/incubator-pulsar/releases/tag/v1.15
1.14 — 2016-08-31
First public release of Pulsar
https://github.com/apache/incubator-pulsar/releases/tag/v1.14