[kinesis-source] Improvements to kinesis-source (#5180)
* Move to using the KCL v2
This moves to using KCL v2 with the new API which supports
"enhanced fan-out". Enhanced fan-out is beneficial as we get more
guranteed capacity and it won't compete with other consumers.
However, support for ehanced fan-out requires pulling in v2 of AWS SDKs.
This isn't a huge problem, as they can be shared, but it does mean
implemting a few shim APIs around credentials.
Most of the business logic remains largely the same, just using the
newer interfaces. Enhanced fan-out can also be disabled and fall back to
a polling model.
Note, this also bumps the kinesis-producer to v0.13, which moves to an
APL license (but has no other changes) to make this nicer to use in
an Apache project without license concerns.
* Add included providers for kinesis auth
This adds two providers for Kinesis auth. First, the
AwsDefaultProviderChain, which just uses the default provider chain
which will pull from env vars, system configuration files, and the
metadata API.
Second, the STSAssumeRole plugin allows us to assume a role (which first
relies on the default provider chain) to run the code under a given role
* add some docs and minor cleanup
14 files changed
tree: 7f2bb1fd1266382c9fe8e4f7bbddad29a4f65bda
- pulsar-io/
