JIRA: HUDI-3730
Hudi support sync to various metastores via different processing framework like Spark, Flink, and Kafka connect.
There are some room for improvement
We need a standard way to run hudi sync. We also need a unified abstraction of XXXSyncTool , XXXSyncClient and XXXSyncConfig to handle supported metastores, like hive metastore, bigquery, datahub, etc.
Below are the proposed key classes to handle the main sync logic. They are extensible for different metastores.
HoodieSyncTool
Renamed from AbstractSyncTool
.
public abstract class HoodieSyncTool implements AutoCloseable { protected HoodieSyncClient syncClient; /** * Sync tool class is the entrypoint to run meta sync. * * @param props A bag of properties passed by users. It can contain all hoodie.* and any other config. * @param hadoopConf Hadoop specific configs. */ public HoodieSyncTool(Properties props, Configuration hadoopConf); public abstract void syncHoodieTable(); public static void main(String[] args) { // instantiate HoodieSyncConfig and concrete sync tool, and run sync. } }
HoodieSyncConfig
public class HoodieSyncConfig extends HoodieConfig { public static class HoodieSyncConfigParams { // POJO class to take command line parameters @Parameter() private String basePath; // common essential parameters public Properties toProps(); } /** * XXXSyncConfig is meant to be created and used by XXXSyncTool exclusively and internally. * * @param props passed from XXXSyncTool. * @param hadoopConf passed from XXXSyncTool. */ public HoodieSyncConfig(Properties props, Configuration hadoopConf); } public class HiveSyncConfig extends HoodieSyncConfig { public static class HiveSyncConfigParams { @Parameter() private String syncMode; // delegate common parameters to other XXXParams class // this overcomes single-inheritance's inconvenience // see https://jcommander.org/#_parameter_delegates @ParametersDelegate() private HoodieSyncConfigParams hoodieSyncConfigParams = new HoodieSyncConfigParams(); public Properties toProps(); } public HoodieSyncConfig(Properties props); }
HoodieSyncClient
Renamed from AbstractSyncHoodieClient
.
public abstract class HoodieSyncClient implements AutoCloseable { // metastore-agnostic APIs }
hoodie.meta.sync.*
hoodie.meta_sync.*
or any other variance to aliashudi-sync
to hudi-meta-sync
(no bundle name change)hoodieSync*
variables or methods to hoodieMetaSync*
database
and table
should not be required by sync tool; they should be inferred from table propertiesMETA_SYNC_BASE_FILE_FORMAT
org.apache.hudi.common.table.HoodieTableConfig.BASE_FILE_FORMAT
META_SYNC_ASSUME_DATE_PARTITION
org.apache.hudi.common.config.HoodieMetadataConfig.ASSUME_DATE_PARTITIONING
META_SYNC_DECODE_PARTITION
org.apache.hudi.common.table.HoodieTableConfig.URL_ENCODE_PARTITIONING
META_SYNC_USE_FILE_LISTING_FROM_METADATA
org.apache.hudi.common.config.HoodieMetadataConfig.ENABLE
The breaking changes should be made at once together with other user-facing config changes at a chosen proper release.
DataSourceOptions
USE_JDBC
and fully adopt SYNC_MODE
HIVE_SYNC_ENDABLED
and related arguments from sync tools and delta streamers. Use SYNC_ENABLED
META_SYNC_BASE_FILE_FORMAT
-> org.apache.hudi.common.table.HoodieTableConfig.BASE_FILE_FORMAT
META_SYNC_ASSUME_DATE_PARTITION
-> org.apache.hudi.common.config.HoodieMetadataConfig.ASSUME_DATE_PARTITIONING
META_SYNC_DECODE_PARTITION
-> org.apache.hudi.common.table.HoodieTableConfig.URL_ENCODE_PARTITIONING
META_SYNC_USE_FILE_LISTING_FROM_METADATA
-> org.apache.hudi.common.config.HoodieMetadataConfig.ENABLE
USE_JDBC
will need to change to set SYNC_MODE=jdbc
--enable-hive-sync
or HIVE_SYNC_ENABLED
will need to drop the argument or config and change to --enable-sync
or SYNC_ENABLED
.DataSourceOptions
for meta sync constants will need to import relevant configs from HoodieSyncConfig
and subclasses.AwsGlueCatalogSyncTool
as sync tool class need to update the class name to AWSGlueCatalogSyncTool