apache /
kudu /
fd7a40367a81555230eac62a1a8d1467433ed100 KUDU-2612: add txn memrowsets to tablet
This patch introduces the ability to insert data into per-transaction
MRSs. When a transaction is started, it creates a new MRS to which
inserts of that transaction can be applied. This "uncommitted
transactional MRS" cannot be inserted to by any other transaction and
cannot be scanned by any scanner. Once the transaction is committed,
i.e. the FINALIZE_COMMIT op is applied for the transaction, the MRS is
moved to a separate set of "committed" transactional MRSs that are
available to be scanned, updated, and flushed. If the transaction is
aborted, the transactional MRS is excised from the list of transactional
MRSs entirely.
When inserting, updating, or scanning rows, ops must now consider all
committed transactional MRSs, in addition to the main, non-transactional
MRS, checking for row presence in any store that is "committed" and
visible to users.
Additionally, when a MRS flush is performed, rather than flushing the
main, non-transactional MRS alone, Kudu will now collect the
non-transactional MRS along with all committed transactional MRSs, and
roll DRSs using the same rowset-merging code that exists for
compactions. The new flushed data will honor each transactions' commit
timestamp rather than the per-row apply timestamps. Existing MRS-related
metrics have been updated to reflect that such flush ops now need to
consider transactional MRSs.
In order to support rebuilding these MRSs when restarting, a new boolean
field is added to the transaction metadata: 'flushed_committed_mrs',
which gets set when the transactional MRS is flushed to disk. If true,
the transaction's write ops do not need to be replayed, since the
transaction does not have an active MRS. Otherwise, the transaction's
write ops do need to be replayed, even if the transaction is persisted
in metadata as being committed.
Additionally, a transaction ID is added to the MutatedStorePB message
stored in the WAL, for the sake of replaying ops that land in committed,
transactional MRSs. Upon replay of such mutations, if the transactional
MRS being mutated to is not active (i.e. it has been flushed), the
mutation op can be ignored.
When starting up, we start transactional MRSs for all transactions that
do not have 'flushed_committed_mrs' set, which indicates that the
transaction's MRS was not flushed before restarting.
NOTE: the locking story here is not complete -- multiple transactions
can write to the same rows without consequence, which violates our row
uniqueness constraints. This will be addressed in future patches.
Change-Id: I1dea4031c0acb875993352d452dc4c233e35a502
Reviewed-on: http://gerrit.cloudera.org:8080/16515
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin <aserbin@cloudera.com>
23 files changed
