tree 471e370b39446215e3635f088e715ed8499175cf
parent dcf7d7edacb8dc7c41903dca94d20ff0809235b9
author ggershinsky <ggershinsky@users.noreply.github.com> 1594022496 +0300
committer Gabor Szadovszky <gabor@apache.org> 1595933542 +0200

PARQUET-1373: Encryption key tools (#615)

* comments
* update key tools
* double wrap for minimizing KMS calls
* Add information about KMS instance ID to footer metadata. Then on file reading, KMS instance ID doesn't have to be provided in properties, but can be read from the metadata.
Add RemoteKmsClient abstract class to assist implementing KMSClients for remote KMSs, that are accessed using URL.
Make DoubleWrappedKeyManager inherit from WrappedKeyManager and make FileKeyManager an abstract class. Add a static factory method to FileKeyManager to initialize an appropriate KSMClient and Key manager.
KMS URL should be specified in properties either directly or in a list. KMS instance ID is either default, or should be specified in properties or read from footer metadata.
* major update - key rotation, crypto factory, etc
* Change caches of EnvelopeKeyManager and EnvelopeKeyRetriever to be per token.
KmsClient is per token and read/write KEK caches too.
Add default token value for InMemoryKMS, which has no tokens.
Use concurrentHashMap for caches with computeIfAbsent.
Add expiration using to the caches - both time-based and on-demand.
On expiration delete the per-token entries from caches.
Add method for cache invalidation per token.
Add abstract methods to be implemented by RemoteKmsClients.
* add in-memory KMS
* Change RemoteKmsClient exceptions to IOException
instead of the higher-level ParquetCryptoRuntimeException.
Change to constant names to uppercase.
* Add sample VaultClient.
* interface changes
* Add okHttp3 dependency for VaultClient sample.
* wrapping changes
* Use JSON serialization for key material.
* separate write and read path, update caching
* improved refactoring
* key rotation improvements
* Add TestPropertiesDrivenEncryption
* get and resfresh token for all KMS clients
* minor changes
* Use ConcurrentHashMap for caches
* caching and store updates
* Rename some encryption/decryption configurations and make the test parameterized
to test combinations of isKeyMaterialExternalStorage, isDoubleWrapping, isWrapLocally.
Add RemoteKmsClient mock for remote wrapping.
* add removeCacheEntriesForAllTokens
* Make common method setCommonKMSProperties and extract classname strings from classes
* Change TestPropertiesDrivenEncryption to accomodate latest API changes.
* Remove StringUtils
* address review comments
* key material documentation
* Boolean objects

Co-authored-by: Maya Anderson <mayaa@il.ibm.com>