docs/source/python/parquet/parquet_encryption.rst

.. Licensed to the Apache Software Foundation (ASF) under one
.. or more contributor license agreements.  See the NOTICE file
.. distributed with this work for additional information
.. regarding copyright ownership.  The ASF licenses this file
.. to you under the Apache License, Version 2.0 (the
.. "License"); you may not use this file except in compliance
.. with the License.  You may obtain a copy of the License at

..   http://www.apache.org/licenses/LICENSE-2.0

.. Unless required by applicable law or agreed to in writing,
.. software distributed under the License is distributed on an
.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
.. KIND, either express or implied.  See the License for the
.. specific language governing permissions and limitations
.. under the License.

.. currentmodule:: pyarrow
.. _parquet_encryption:


Parquet Modular Encryption (Columnar Encryption)
================================================

Columnar encryption is supported for Parquet files in C++ starting from
Apache Arrow 4.0.0 and in PyArrow starting from Apache Arrow 6.0.0.

Parquet uses the envelope encryption practice, where file parts are encrypted
with "data encryption keys" (DEKs), and the DEKs are encrypted with "master
encryption keys" (MEKs). The DEKs are randomly generated by Parquet for each
encrypted file/column. The MEKs are generated, stored and managed in a Key
Management Service (KMS) of user’s choice.

Reading and writing encrypted Parquet files involves passing file encryption
and decryption properties to :class:`~pyarrow.parquet.ParquetWriter` and to
:class:`~pyarrow.parquet.ParquetFile`, respectively.

Writing an encrypted Parquet file:

.. code-block:: python

   >>> import pyarrow.parquet as pq
   >>> encryption_properties = crypto_factory.file_encryption_properties(  # doctest: +SKIP
   ...                                  kms_connection_config, encryption_config)
   >>> with pq.ParquetWriter(filename, schema,  # doctest: +SKIP
   ...                      encryption_properties=encryption_properties) as writer:
   ...    writer.write_table(table)

Reading an encrypted Parquet file:

.. code-block:: python

   >>> decryption_properties = crypto_factory.file_decryption_properties(  # doctest: +SKIP
   ...                                                  kms_connection_config)
   >>> parquet_file = pq.ParquetFile(filename,  # doctest: +SKIP
   ...                               decryption_properties=decryption_properties)


In order to create the encryption and decryption properties, a
:class:`pyarrow.parquet.encryption.CryptoFactory` should be created and
initialized with KMS Client details, as described below.


KMS Client
~~~~~~~~~~

The master encryption keys should be kept and managed in a production-grade
Key Management System (KMS), deployed in the user's organization. Using Parquet
encryption requires implementation of a client class for the KMS server.
Any KmsClient implementation should implement the informal interface
defined by :class:`pyarrow.parquet.encryption.KmsClient` as follows:

.. code-block:: python

   >>> import pyarrow.parquet.encryption as pe
   >>> class MyKmsClient(pe.KmsClient):
   ...
   ...    """An example KmsClient implementation skeleton"""
   ...    def __init__(self, kms_connection_configuration):
   ...       pe.KmsClient.__init__(self)
   ...       # Any KMS-specific initialization based on
   ...       # kms_connection_configuration comes here
   ...
   ...    def wrap_key(self, key_bytes, master_key_identifier):
   ...       wrapped_key = ... # call KMS to wrap key_bytes with key specified by
   ...                         # master_key_identifier
   ...       return wrapped_key
   ...
   ...    def unwrap_key(self, wrapped_key, master_key_identifier):
   ...       key_bytes = ... # call KMS to unwrap wrapped_key with key specified by
   ...                       # master_key_identifier
   ...       return key_bytes

The concrete implementation will be loaded at runtime by a factory function
provided by the user. This factory function will be used to initialize the
:class:`pyarrow.parquet.encryption.CryptoFactory` for creating file encryption
and decryption properties.

For example, in order to use the ``MyKmsClient`` defined above:

.. code-block:: python

   >>> def kms_client_factory(kms_connection_configuration):
   ...    return MyKmsClient(kms_connection_configuration)

   >>> crypto_factory = pe.CryptoFactory(kms_client_factory)

An :download:`example <../../../../python/examples/parquet_encryption/sample_vault_kms_client.py>`
of such a class for an open source
`KMS <https://www.vaultproject.io/api/secret/transit>`_ can be found in the Apache
Arrow GitHub repository. The production KMS client should be designed in
cooperation with an organization's security administrators, and built by
developers with experience in access control management. Once such a class is
created, it can be passed to applications via a factory method and leveraged
by general PyArrow users as shown in the encrypted parquet write/read sample
above.

KMS connection configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Configuration of connection to KMS (:class:`pyarrow.parquet.encryption.KmsConnectionConfig`
used when creating file encryption and decryption properties) includes the
following options:

* ``kms_instance_url``, URL of the KMS instance.
* ``kms_instance_id``, ID of the KMS instance that will be used for encryption
  (if multiple KMS instances are available).
* ``key_access_token``, authorization token that will be passed to KMS.
* ``custom_kms_conf``, a string dictionary with KMS-type-specific configuration.

Encryption configuration
~~~~~~~~~~~~~~~~~~~~~~~~

:class:`pyarrow.parquet.encryption.EncryptionConfiguration` (used when
creating file encryption properties) includes the following options:

* ``footer_key``, the ID of the master key for footer encryption/signing.
* ``column_keys``, which columns to encrypt with which key. Dictionary with
  master key IDs as the keys, and column name lists as the values,
  e.g. ``{key1: [col1, col2], key2: [col3]}``. See notes on nested fields below.
* ``uniform_encryption``, whether to encrypt the footer and all columns with
  the same ``footer_key``, instead of specifying ``column_keys``
  individually. Cannot be used together with ``column_keys``.
* ``encryption_algorithm``, the Parquet encryption algorithm.
  Can be ``AES_GCM_V1`` (default) or ``AES_GCM_CTR_V1``.
* ``plaintext_footer``, whether to write the file footer in plain text (otherwise it is encrypted).
* ``double_wrapping``, whether to use double wrapping - where data encryption keys (DEKs)
  are encrypted with key encryption keys (KEKs), which in turn are encrypted
  with master encryption keys (MEKs). If set to ``false``, single wrapping is
  used - where DEKs are encrypted directly with MEKs.
* ``cache_lifetime``, the lifetime of cached entities (key encryption keys,
  local wrapping keys, KMS client objects) represented as a ``datetime.timedelta``.
* ``internal_key_material``, whether to store key material inside Parquet file footers;
  this mode doesn’t produce additional files. If set to ``false``, key material is
  stored in separate files in the same folder, which enables key rotation for
  immutable Parquet files.
* ``data_key_length_bits``, the length of data encryption keys (DEKs), randomly
  generated by Parquet key management tools. Can be 128, 192 or 256 bits.

.. note::
   When ``double_wrapping`` is true, Parquet implements a "double envelope
   encryption" mode that minimizes the interaction of the program with a KMS
   server. In this mode, the DEKs are encrypted with "key encryption keys"
   (KEKs, randomly generated by Parquet). The KEKs are encrypted with "master
   encryption keys" (MEKs) in the KMS; the result and the KEK itself are
   cached in the process memory.

An example encryption configuration:

.. code-block:: python

   >>> encryption_config = pe.EncryptionConfiguration(
   ...    footer_key="footer_key_name",
   ...    column_keys={
   ...       "column_key_name": ["Column1", "Column2"],
   ...    },
   ... )

.. note::

   Columns with nested fields (struct or map data types) can be encrypted as a whole, or only
   individual fields. Configure an encryption key for the root column name to encrypt all nested
   fields with this key, or configure a key for individual leaf nested fields.

   Conventionally, the key and value fields of a map column ``m`` have the names
   ``m.key_value.key`` and ``m.key_value.value``, respectively.
   An inner field ``f`` of a struct column ``s`` has the name ``s.f``.

   With above example, *all* inner fields are encrypted with the same key by configuring that key
   for column ``m`` and ``s``, respectively.

An example encryption configuration for columns with nested fields, where
all columns are encrypted with the same key identified by ``column_key_id``:

.. code-block:: python

   >>> import pyarrow as pa
   >>> schema = pa.schema([
   ...   ("MapColumn", pa.map_(pa.string(), pa.int32())),
   ...   ("StructColumn", pa.struct([("f1", pa.int32()), ("f2", pa.string())])),
   ... ])

   >>> encryption_config = pe.EncryptionConfiguration(
   ...    footer_key="footer_key_name",
   ...    column_keys={
   ...       "column_key_id": [ "MapColumn", "StructColumn" ],
   ...    },
   ... )

An example encryption configuration for columns with nested fields, where
some inner fields are encrypted with the same key identified by ``column_key_id``:

.. code-block:: python

   >>> schema = pa.schema([
   ...   ("MapColumn", pa.map_(pa.string(), pa.int32())),
   ...   ("StructColumn", pa.struct([("f1", pa.int32()), ("f2", pa.string())])),
   ... ])

   >>> encryption_config = pe.EncryptionConfiguration(
   ...    footer_key="footer_key_name",
   ...    column_keys={
   ...       "column_key_id": [ "MapColumn.key_value.value", "StructColumn.f1" ],
   ...    },
   ... )

Decryption configuration
~~~~~~~~~~~~~~~~~~~~~~~~

:class:`pyarrow.parquet.encryption.DecryptionConfiguration` (used when creating
file decryption properties) is optional and it includes the following options:

* ``cache_lifetime``, the lifetime of cached entities (key encryption keys, local
  wrapping keys, KMS client objects) represented as a ``datetime.timedelta``.
* ``read_kms_url``, whether the KMS instance URL may be read from the key material
  of the file being read, when it is not set in the ``KmsConnectionConfig``. This
  defaults to ``False``, and should only be enabled when the KMS implementation
  validates the URL it receives, to ensure a KMS access token isn't sent to a
  malicious URL.

External key material and key rotation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When ``internal_key_material=False`` is set on ``EncryptionConfiguration``,
key material is stored in a separate file next to the Parquet file instead of
in its footer.

Storing key material externally is what enables key rotation:
:meth:`~pyarrow.parquet.encryption.CryptoFactory.rotate_master_keys` re-wraps the data encryption keys of
a file that uses external key material using new master keys and overwrites the external
key material file, without changing the Parquet file itself:

.. code-block:: python

   >>> crypto_factory.rotate_master_keys(  # doctest: +SKIP
   ...     kms_connection_config, parquet_file_path="table.parquet",
   ... )

``rotate_master_keys`` also accepts ``read_kms_url``, which behaves like the
``DecryptionConfiguration`` option of the same name when the existing key material is
read. The key material written by key rotation always uses the connection properties
from the ``KmsConnectionConfig`` that is passed in.

Direct Key Encryption (without KMS)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

:func:`pyarrow.parquet.encryption.create_encryption_properties` and
:func:`pyarrow.parquet.encryption.create_decryption_properties` build
encryption/decryption properties directly from a plaintext key, bypassing
:class:`~pyarrow.parquet.encryption.CryptoFactory` and the KMS-based flow.

.. note::
   Only uniform encryption (a single key for the footer and all columns) is
   supported by these functions. For per-column keys, use the
   ``CryptoFactory``/``EncryptionConfiguration`` flow described above.

.. code-block:: python

   >>> props = pe.create_encryption_properties(  # doctest: +SKIP
   ...     footer_key=b'0123456789abcdef',
   ...     aad_prefix=b'table_id',
   ...     store_aad_prefix=False,
   ... )
   >>> pq.write_table(table, 'encrypted.parquet', encryption_properties=props)  # doctest: +SKIP

   >>> decryption_props = pe.create_decryption_properties(  # doctest: +SKIP
   ...     footer_key=b'0123456789abcdef',
   ...     aad_prefix=b'table_id',
   ... )
   >>> pq.read_table('encrypted.parquet', decryption_properties=decryption_props)  # doctest: +SKIP