src/main/asciidoc/_chapters/inmemory_compaction.adoc - hbase - Git at Google

 ////
 /**
  *
  * Licensed to the Apache Software Foundation (ASF) under one
  * or more contributor license agreements.  See the NOTICE file
  * distributed with this work for additional information
  * regarding copyright ownership.  The ASF licenses this file
  * to you under the Apache License, Version 2.0 (the
  * "License"); you may not use this file except in compliance
  * with the License.  You may obtain a copy of the License at
  *
  *     http://www.apache.org/licenses/LICENSE-2.0
  *
  * Unless required by applicable law or agreed to in writing, software
  * distributed under the License is distributed on an "AS IS" BASIS,
  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
 ////

 [[inmemory_compaction]]
 = In-memory Compaction
 :doctype: book
 :numbered:
 :toc: left
 :icons: font
 :experimental:

 [[imc.overview]]
 == Overview

 In-memory Compaction (A.K.A Accordion) is a new feature in hbase-2.0.0.
 It was first introduced on the Apache HBase Blog at
 link:https://blogs.apache.org/hbase/entry/accordion-hbase-breathes-with-in[Accordion: HBase Breathes with In-Memory Compaction].
 Quoting the blog:
 ____
 Accordion reapplies the LSM principal [_Log-Structured-Merge Tree_, the design pattern upon which HBase is based] to MemStore, in order to eliminate redundancies and other overhead while the data is still in RAM.  Doing so decreases the frequency of flushes to HDFS, thereby reducing the write amplification and the overall disk footprint. With less flushes, the write operations are stalled less frequently as the MemStore overflows, therefore the write performance is improved. Less data on disk also implies less pressure on the block cache, higher hit rates, and eventually better read response times. Finally, having less disk writes also means having less compaction happening in the background, i.e., less cycles are stolen from productive (read and write) work. All in all, the effect of in-memory compaction can be envisioned as a catalyst that enables the system move faster as a whole.
 ____

 A developer view is available at
 link:https://blogs.apache.org/hbase/entry/accordion-developer-view-of-in[Accordion: Developer View of In-Memory Compaction].

 In-memory compaction works best when high data churn; overwrites or over-versions
 can be eliminated while the data is still in memory. If the writes are all uniques,
 it may drag write throughput (In-memory compaction costs CPU). We suggest you test
 and compare before deploying to production.

 In this section we describe how to enable Accordion and the available configurations.

 == Enabling

 To enable in-memory compactions, set the _IN_MEMORY_COMPACTION_ attribute
 on per column family where you want the behavior. The _IN_MEMORY_COMPACTION_
 attribute can have one of four values.

  * _NONE_: No in-memory compaction.
  * _BASIC_: Basic policy enables flushing and keeps a pipeline of flushes until we trip the pipeline maximum threshold and then we flush to disk. No in-memory compaction but can help throughput as data is moved from the profligate, native ConcurrentSkipListMap data-type to more compact (and efficient) data types.
  * _EAGER_: This is _BASIC_ policy plus in-memory compaction of flushes (much like the on-disk compactions done to hfiles); on compaction we apply on-disk rules eliminating versions, duplicates, ttl'd cells, etc.
  * _ADAPTIVE_: Adaptive compaction adapts to the workload. It applies either index compaction or data compaction based on the ratio of duplicate cells in the data.  Experimental.

 To enable _BASIC_ on the _info_ column family in the table _radish_, add the attribute to the _info_ column family:
 [source,ruby]
 ----
 hbase(main):003:0> alter 'radish', {NAME => 'info', IN_MEMORY_COMPACTION => 'BASIC'}
 Updating all regions with the new schema...
 All regions updated.
 Done.
 Took 1.2413 seconds
 hbase(main):004:0> describe 'radish'
 Table radish is DISABLED
 radish
 COLUMN FAMILIES DESCRIPTION
 {NAME => 'info', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536', METADATA => {
 'IN_MEMORY_COMPACTION' => 'BASIC'}}
 1 row(s)
 Took 0.0239 seconds
 ----
 Note how the IN_MEMORY_COMPACTION attribute shows as part of the _METADATA_ map.

 There is also a global configuration, _hbase.hregion.compacting.memstore.type_ which you can set in your _hbase-site.xml_ file. Use it to set the
 default on creation of a new table (On creation of a column family Store, we look first to the column family configuration looking for the
 _IN_MEMORY_COMPACTION_ setting, and if none, we then consult the _hbase.hregion.compacting.memstore.type_ value using its content; default is
 _BASIC_).

 By default, new hbase system tables will have _BASIC_ in-memory compaction set.  To specify otherwise,
 on new table-creation, set _hbase.hregion.compacting.memstore.type_ to _NONE_ (Note, setting this value
 post-creation of system tables will not have a retroactive effect; you will have to alter your tables
 to set the in-memory attribute to _NONE_).

 When an in-memory flush happens is calculated by dividing the configured region flush size (Set in the table descriptor
 or read from _hbase.hregion.memstore.flush.size_) by the number of column families and then multiplying by
 _hbase.memstore.inmemoryflush.threshold.factor_. Default is 0.014.

 The number of flushes carried by the pipeline is monitored so as to fit within the bounds of memstore sizing
 but you can also set a maximum on the number of flushes total by setting
 _hbase.hregion.compacting.pipeline.segments.limit_. Default is 2.

 When a column family Store is created, it says what memstore type is in effect. As of this writing
 there is the old-school _DefaultMemStore_ which fills a _ConcurrentSkipListMap_ and then flushes
 to disk or the new _CompactingMemStore_ that is the implementation that provides this new
 in-memory compactions facility. Here is a log-line from a RegionServer that shows a column
 family Store named _family_ configured to use a _CompactingMemStore_:
 ----
 Note how the IN_MEMORY_COMPACTION attribute shows as part of the _METADATA_ map.
 2018-03-30 11:02:24,466 INFO  [Time-limited test] regionserver.HStore(325): Store=family,  memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=10
 ----

 Enable TRACE-level logging on the CompactingMemStore class (_org.apache.hadoop.hbase.regionserver.CompactingMemStore_) to see detail on its operation.
	////
	/**
	*
	* Licensed to the Apache Software Foundation (ASF) under one
	* or more contributor license agreements. See the NOTICE file
	* distributed with this work for additional information
	* regarding copyright ownership. The ASF licenses this file
	* to you under the Apache License, Version 2.0 (the
	* "License"); you may not use this file except in compliance
	* with the License. You may obtain a copy of the License at
	*
	* http://www.apache.org/licenses/LICENSE-2.0
	*
	* Unless required by applicable law or agreed to in writing, software
	* distributed under the License is distributed on an "AS IS" BASIS,
	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	* See the License for the specific language governing permissions and
	* limitations under the License.
	*/
	////

	[[inmemory_compaction]]
	= In-memory Compaction
	:doctype: book
	:numbered:
	:toc: left
	:icons: font
	:experimental:

	[[imc.overview]]
	== Overview

	In-memory Compaction (A.K.A Accordion) is a new feature in hbase-2.0.0.
	It was first introduced on the Apache HBase Blog at
	link:https://blogs.apache.org/hbase/entry/accordion-hbase-breathes-with-in[Accordion: HBase Breathes with In-Memory Compaction].
	Quoting the blog:
	____
	Accordion reapplies the LSM principal [_Log-Structured-Merge Tree_, the design pattern upon which HBase is based] to MemStore, in order to eliminate redundancies and other overhead while the data is still in RAM. Doing so decreases the frequency of flushes to HDFS, thereby reducing the write amplification and the overall disk footprint. With less flushes, the write operations are stalled less frequently as the MemStore overflows, therefore the write performance is improved. Less data on disk also implies less pressure on the block cache, higher hit rates, and eventually better read response times. Finally, having less disk writes also means having less compaction happening in the background, i.e., less cycles are stolen from productive (read and write) work. All in all, the effect of in-memory compaction can be envisioned as a catalyst that enables the system move faster as a whole.
	____

	A developer view is available at
	link:https://blogs.apache.org/hbase/entry/accordion-developer-view-of-in[Accordion: Developer View of In-Memory Compaction].

	In-memory compaction works best when high data churn; overwrites or over-versions
	can be eliminated while the data is still in memory. If the writes are all uniques,
	it may drag write throughput (In-memory compaction costs CPU). We suggest you test
	and compare before deploying to production.

	In this section we describe how to enable Accordion and the available configurations.

	== Enabling

	To enable in-memory compactions, set the _IN_MEMORY_COMPACTION_ attribute
	on per column family where you want the behavior. The _IN_MEMORY_COMPACTION_
	attribute can have one of four values.

	* _NONE_: No in-memory compaction.
	* _BASIC_: Basic policy enables flushing and keeps a pipeline of flushes until we trip the pipeline maximum threshold and then we flush to disk. No in-memory compaction but can help throughput as data is moved from the profligate, native ConcurrentSkipListMap data-type to more compact (and efficient) data types.
	* _EAGER_: This is _BASIC_ policy plus in-memory compaction of flushes (much like the on-disk compactions done to hfiles); on compaction we apply on-disk rules eliminating versions, duplicates, ttl'd cells, etc.
	* _ADAPTIVE_: Adaptive compaction adapts to the workload. It applies either index compaction or data compaction based on the ratio of duplicate cells in the data. Experimental.

	To enable _BASIC_ on the _info_ column family in the table _radish_, add the attribute to the _info_ column family:
	[source,ruby]
	----
	hbase(main):003:0> alter 'radish', {NAME => 'info', IN_MEMORY_COMPACTION => 'BASIC'}
	Updating all regions with the new schema...
	All regions updated.
	Done.
	Took 1.2413 seconds
	hbase(main):004:0> describe 'radish'
	Table radish is DISABLED
	radish
	COLUMN FAMILIES DESCRIPTION
	{NAME => 'info', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536', METADATA => {
	'IN_MEMORY_COMPACTION' => 'BASIC'}}
	1 row(s)
	Took 0.0239 seconds
	----
	Note how the IN_MEMORY_COMPACTION attribute shows as part of the _METADATA_ map.

	There is also a global configuration, _hbase.hregion.compacting.memstore.type_ which you can set in your _hbase-site.xml_ file. Use it to set the
	default on creation of a new table (On creation of a column family Store, we look first to the column family configuration looking for the
	_IN_MEMORY_COMPACTION_ setting, and if none, we then consult the _hbase.hregion.compacting.memstore.type_ value using its content; default is
	_BASIC_).

	By default, new hbase system tables will have _BASIC_ in-memory compaction set. To specify otherwise,
	on new table-creation, set _hbase.hregion.compacting.memstore.type_ to _NONE_ (Note, setting this value
	post-creation of system tables will not have a retroactive effect; you will have to alter your tables
	to set the in-memory attribute to _NONE_).

	When an in-memory flush happens is calculated by dividing the configured region flush size (Set in the table descriptor
	or read from _hbase.hregion.memstore.flush.size_) by the number of column families and then multiplying by
	_hbase.memstore.inmemoryflush.threshold.factor_. Default is 0.014.

	The number of flushes carried by the pipeline is monitored so as to fit within the bounds of memstore sizing
	but you can also set a maximum on the number of flushes total by setting
	_hbase.hregion.compacting.pipeline.segments.limit_. Default is 2.

	When a column family Store is created, it says what memstore type is in effect. As of this writing
	there is the old-school _DefaultMemStore_ which fills a _ConcurrentSkipListMap_ and then flushes
	to disk or the new _CompactingMemStore_ that is the implementation that provides this new
	in-memory compactions facility. Here is a log-line from a RegionServer that shows a column
	family Store named _family_ configured to use a _CompactingMemStore_:
	----
	Note how the IN_MEMORY_COMPACTION attribute shows as part of the _METADATA_ map.
	2018-03-30 11:02:24,466 INFO [Time-limited test] regionserver.HStore(325): Store=family, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=10
	----

	Enable TRACE-level logging on the CompactingMemStore class (_org.apache.hadoop.hbase.regionserver.CompactingMemStore_) to see detail on its operation.