tree 0e25668b27c0b3c670b69bb5a882d3ad55b9f7aa
parent 26589491d13641a167ee3cceb65b7fec8eb37e9c
author David Kjerrumgaard <35466513+david-streamlio@users.noreply.github.com> 1603072584 -0700
committer GitHub <noreply@github.com> 1603072584 +0800

Refactored JCloud Tiered Storage (#6335)

Motivation

In order to facilitate the support of additional JClouds-supported providers, we first needed to clean up the existing code, as there were a lot of if/then/else constructs throughout the code that were based on the assumption that we either supported AWS S3 or Google Cloud Storage. I didn't want to keep adding else if's to these code blocks for every new provider we add, so I decided to refactor the code to make it a bit cleaner

Modifications

in addition to being home for most of the aforementioned if/then/else blocks, the BlobStoreManagedLedgerOffloader class had multiple responsibilities in addition to providing an implementation for the LedgerOffloader interface. My goal was to simplify this class such that its only responsibility was to implement the LedgerOffloader interface.

The other major change was the addition of the JCloudBlobStoreProvider enum, which implements 3 interfaces that allow for it to handle the provider specific logic for things such as acquiring the credentials, validating the configuration, and creating a provider-specific instance of BlobStore.

Result

After this change, we will be able to easily add support for additional JClouds-supported providers by simply adding new elements to the JCloudBlobStoreProvider Enums since the other logic has been isolated and is not vendor specific.

See #2865 for more details

* Refactored JCloud Tiered Storage

* Refactored JCloud Tiered Storage

* Added missing import statements

* Refactored JCloud Tiered Storage

* Refactored JCloud Tiered Storage

* Added missing import statements

* Refactored JCloud Tiered Storage

* Refactored JCloud Tiered Storage

* Added missing import statements

* fix test

* add test logs

* fix logs

* fix test

* fix

* fix

* fix broker log

* fix configuration

* fix

* fix

* fix

* fix get BlobStore

* repair test presto query tiered storage data

* fix test

* fix test

* fix test TestPrestoQueryTieredStorage

* fix

* fix

Co-authored-by: Sijie Guo <sijie@apache.org>
Co-authored-by: gaoran10 <gaoran_10@126.com>
Co-authored-by: xiaolong.ran <rxl@apache.org>