The Gobblin Compliance module allows for data purging to meet regulatory compliance requirements. The module includes functionality for purging datasets and the associated operational support in production.
The purging is performed using Hive meaning that purging of datasets is supported in any format that Hive can read from and write to, including for example ORC and Parquet. Further the purger is built on top of the Gobblin framework which means that the fault-tolerance, scalability and flexibility provided by Gobblin is taken full advantage of.
As an example, let us assume that regulation requires that once a guest has checked out of a hotel, certain guest data needs to be purged within a certain number of days after the guest leaves. Hence, the goal is to purge the hotel's datasets of data associated with their guests after they have left in order to meet regulatory compliance requirements.
Hive databases and tables are setup to be purged by following these steps:
For example, in order to purge a Hive table named tracking.event
, these properties are specified:
gobblin.compliance.dataset.whitelist=tracking.event
dataset.descriptor
to the tracking.event Hive table to specify the compliance field to match in the table as an escaped JSON (since it has to be a valid string): {\"complianceSpec\" : {\"identifierField\" : \"metadata.guestid\" }}
dataset.descriptor.fieldPath=complianceSpec.identifierField
gobblin.compliance.complianceIdTable=u_purger.guestIds
gobblin.compliance.complianceId=guestId
With these properties in place, a Gobblin job can be setup to purge the table. The work unit for the purger is an individual table partition. Hence the purger will iterate over all the partitions in the table, and purge each partition individually, processing as many partitions in parallel as specified (by the property ‘gobblin.compliance.purger.maxWorkunits’, which defaults to 5).
Configuration options for the Hive Purger include
Property | Description |
---|---|
gobblin.compliance.dataset.whitelist | The list of databases/tables to purge, comma-separated |
dataset.descriptor.fieldPath | The JSON field path specifying the compliance field |
gobblin.compliance.complianceIdTable | The table containing the list of ids whose data needs to be purged |
gobblin.compliance.complianceId | The name of the id column in the complianceIdTable to match against |
gobblin.compliance.purger.maxWorkunits | The number of partitions to purge in parallel |
gobblin.compliance.purger.policy.class | The policy class that specifies the criteria for purging, defaults to HivePurgerPolicy |
gobblin.compliance.purger.commit.policy.class | The policy class that specifies the criteria for committing a purged dataset, defaults to HivePurgerCommitPolicy |
The Developer Guide further describes the design of the module.