commit | d52a1b0f02e3eee94dc1465fd3d0fa101897ab99 | [log] [tgz] |
---|---|---|
author | Enrico Olivelli <eolivelli@gmail.com> | Mon Mar 15 08:33:41 2021 +0100 |
committer | GitHub <noreply@github.com> | Mon Mar 15 15:33:41 2021 +0800 |
tree | 3d09bc84eb83f2d8afafb8dcb6f974c4bb849a26 | |
parent | 5190af3ec8a202701620ba9e4a8c9d4ed7558118 [diff] |
Pulsar IO - KafkaSource - allow to manage Avro Encoded messages (#9448) ### Motivation Currently KafkaSource allows only to deal with strings and byte arrays, it does not support records with Schema. In Kafka we have the ability to encode messages using Avro and there is a Schema Registry (by Confluent®) ### Modifications Summary of changes: - allow current KafkaSource (`KafkaBytesSource`) to deal with `io.confluent.kafka.serializers.KafkaAvroDeserializer ` and copy the raw bytes to the Pulsar topic, setting appropriately the Schema - this source support Schema Evolution end-to-end (i.e. add fields to the original schema in the Kafka world, and see the new fields in the Pulsar topic, without any reconfiguration or restart) - add Confluent® Schema Registry Client to the Kafka Connector NAR, the license is compatible with Apache 2 license and we can redistribute it - the configuration of the Schema Registry Client is done done in the consumerProperties property of the source (usually you add schema.registry.url) - add integration tests with Kafka and Schema Registry ### Verifying this change The patch introduces new integration tests. The integration tests launch a Kafka Container and also a Confluent Schema Registry Container
Pulsar is a distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API.
Learn more about Pulsar at https://pulsar.apache.org
This repository is the main repository of Apache Pulsar. Pulsar PMC also maintains other repositories for components in the Pulsar ecosystem, including connectors, adapters, and other language clients.
Requirements:
Compile and install:
$ mvn install -DskipTests
mvn install -Pcore-modules
Run Unit Tests:
$ mvn test
Run Individual Unit Test:
$ cd module-name (e.g: pulsar-client) $ mvn test -Dtest=unit-test-name (e.g: ConsumerBuilderImplTest)
Run Selected Test packages:
$ cd module-name (e.g: pulsar-broker) $ mvn test -pl module-name -Dinclude=org/apache/pulsar/**/*.java
Start standalone Pulsar service:
$ bin/pulsar standalone
Check https://pulsar.apache.org for documentation and examples.
Apache Pulsar is using lombok so you have to ensure your IDE setup with required plugins.
Open Annotation Processors Settings dialog box by going to Settings -> Build, Execution, Deployment -> Compiler -> Annotation Processors
.
Select the following buttons:
Set the generated source directories to be equal to the Maven directories:
Click “OK”.
Install the lombok plugin in intellij.
When working on the Pulsar core modules in IntelliJ, reduce the number of active projects in IntelliJ to speed up IDE actions and reduce unrelated IDE warnings.
Run the “Generate Sources and Update Folders For All Projects” action from the Maven UI toolbar. You can also find the action by the name in the IntelliJ “Search Everywhere” window that gets activated by pressing the Shift key twice. Running the action takes about 10 minutes for all projects. This is faster when the “core-modules” profile is the only active profile.
In the case of compilation errors with missing Protobuf classes, ensure to run the “Generate Sources and Update Folders For All Projects” action.
All of the Pulsar source code doesn't compile properly in IntelliJ and there are compilation errors.
mvn test -Dtest=TestClassName
command.Follow the instructions here to configure your Eclipse setup.
Refer to the docs README.
Name | Scope | |||
---|---|---|---|---|
users@pulsar.apache.org | User-related discussions | Subscribe | Unsubscribe | Archives |
dev@pulsar.apache.org | Development-related discussions | Subscribe | Unsubscribe | Archives |
Pulsar slack channel at https://apache-pulsar.slack.com/
You can self-register at https://apache-pulsar.herokuapp.com/
Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0
This distribution includes cryptographic software. The country in which you currently reside may have restrictions on the import, possession, use, and/or re-export to another country, of encryption software. BEFORE using any encryption software, please check your country's laws, regulations and policies concerning the import, possession, or use, and re-export of encryption software, to see if this is permitted. See http://www.wassenaar.org/ for more information.
The U.S. Government Department of Commerce, Bureau of Industry and Security (BIS), has classified this software as Export Commodity Control Number (ECCN) 5D002.C.1, which includes information security software using or performing cryptographic functions with asymmetric algorithms. The form and manner of this Apache Software Foundation distribution makes it eligible for export under the License Exception ENC Technology Software Unrestricted (TSU) exception (see the BIS Export Administration Regulations, Section 740.13) for both object code and source code.
The following provides more details on the included cryptographic software: Pulsar uses the SSL library from Bouncy Castle written by http://www.bouncycastle.org.