JIRA: https://issues.apache.org/jira/browse/HUDI-3217
Please keep the status updated in
rfc/README.md
.
Avro historically has been a centerpiece of the Hudi architecture: it's a default representation that many components expect when dealing with records (during merge, column value extractions, writing into storage, etc).
While having a single format of the record representation is certainly making implementation of some components simpler, it bears unavoidable performance penalty of de-/serialization loop: every record handled by Hudi has to be converted from (low-level) engine-specific representation (Row
for Spark, RowData
for Flink, ArrayWritable
for Hive) into intermediate one (Avro), with some operations (like clustering, compaction) potentially incurring this penalty multiple times (on read- and write-paths).
As such, goal of this effort is to remove the need of conversion from engine-specific internal representations to Avro while handling records.
Historically, Avro has settled in as de-facto intermediate representation of the record's payload since the early days of Hudi. As project matured and the scale of the installations grew, necessity to convert into an intermediate representation quickly become a noticeable bottleneck in terms of performance of critical Hudi flows.
At the center of it is the hierarchy of HoodieRecordPayload
s, which is used to hold individual record's payload providing an APIs like preCombine
, combineAndGetUpdateValue
to combine it with other record using some user-defined semantic.
To achieve stated goals of avoiding unnecessary conversions into intermediate representation (Avro), existing Hudi workflows operating on individual records will have to be refactored and laid out in a way that would be unassuming about internal representation of the record, ie code should be working w/ a record as an opaque object: exposing certain API to access crucial data (precombine, primary, partition keys, etc), but not providing the access to the raw payload.
Having existing workflows re-structured in such a way around a record being an opaque object, would allow us to encapsulate internal representation of the record w/in its class hierarchy, which in turn would allow for us to hold engine-specific (Spark, Flink, etc) representations of the records w/o exposing purely engine-agnostic components to it.
Following (high-level) steps are proposed:
HoodieRecord
to become a standardized API of interacting with a single record, that will beHoodieRecordPayload
getPartitionKey
, getRecordKey
, etc)HoodieRecordPayload
into a standalone, stateless component (engine). Such component will beHoodieRecordPayload
abstractionPhasing out usage of HoodieRecordPayload
will also bring the benefit of avoiding to use Java reflection in the hot-path, which is known to have poor performance (compared to non-reflection based instantiation).
Stateless component interface providing for API Combining Records will look like following:
interface HoodieRecordCombiningEngine { default HoodieRecord precombine(HoodieRecord older, HoodieRecord newer) { if (spark) { precombineSpark((SparkHoodieRecord) older, (SparkHoodieRecord) newer); } else if (flink) { // precombine for Flink } } /** * Spark-specific implementation */ SparkHoodieRecord precombineSpark(SparkHoodieRecord older, SparkHoodieRecord newer); // ... }
Where user can provide their own subclass implementing such interface for the engines of interest.
HoodieRecordPayload
to HoodieRecordCombiningEngine
To warrant backward-compatibility (BWC) on the code-level with already created subclasses of HoodieRecordPayload
currently already used in production by Hudi users, we will provide a BWC-bridge in the form of instance of HoodieRecordCombiningEngine
, that will be using user-defined subclass of HoodieRecordPayload
to combine the records.
Leveraging such bridge will make provide for seamless BWC migration to the 0.11 release, however will be removing the performance benefit of this refactoring, since it would unavoidably have to perform conversion to intermediate representation (Avro). To realize full-suite of benefits of this refactoring, users will have to migrate their merging logic out of HoodieRecordPayload
subclass and into new HoodieRecordCombiningEngine
implementation.
As was called out prior to achieve the goal of being able to sustain engine-internal representations being held by HoodieRecord
class w/o compromising major components' neutrality (ie being engine-agnostic), such components directly interacting w/ records' payloads today will have to be refactored to instead interact w/ standardized HoodieRecord
s API.
Following major components will be refactored:
HoodieWriteHandle
s will beHoodieRecord
instead of raw Avro payload (avoiding Avro conversion)HoodieRecord
as is to FileWriter
HoodieFileWriter
s will beHoodieRecord
HoodieRealtimeRecordReader
sHoodieRecord
instead of raw Avro payloadHoodieRecordPayload
, and instead subclass newly created interface HoodieRecordCombiningEngine
to get full-suite of performance benefitsHoodieRecordPayload
for merging will be marked as deprecated in 0.11, and subsequently removed in 0.1xThis refactoring will not be modifying any existing Hudi semantics other than the aforementioned, and as such to guarantee preservation of the logical correctness of the many flows that will be affected by the refactoring we will rely on the existing set of test-suites.
Nevertheless, we will run corresponding set of benchmarks stressing the flows being affected by the refactoring to validate that there are considerable performance advantage of abandoning conversion into intermediate representation completely.