HBASE Changelog

Release 2.1.0 - Unreleased (as of 2018-07-10)

INCOMPATIBLE CHANGES:

JIRA	Summary	Priority	Component
HBASE-20691	Storage policy should allow deferring to HDFS	Blocker	Filesystem Integration, wal
HBASE-20270	Turn off command help that follows all errors in shell	Major	shell
HBASE-20501	Change the Hadoop minimum version to 2.7.1	Blocker	community, documentation
HBASE-20406	HBase Thrift HTTP - Shouldn't handle TRACE/OPTIONS methods	Major	security, Thrift
HBASE-20159	Support using separate ZK quorums for client	Major	Client, Operability, Zookeeper
HBASE-20148	Make serial replication as a option for a peer instead of a table	Major	Replication

NEW FEATURES:

JIRA	Summary	Priority	Component
HBASE-20833	Modify pre-upgrade coprocessor validator to support table level coprocessors	Major	Coprocessors
HBASE-15809	Basic Replication WebUI	Critical	Replication, UI
HBASE-19735	Create a minimal “client” tarball installation	Major	build, Client
HBASE-20656	Validate pre-2.0 coprocessors against HBase 2.0+	Major	tooling
HBASE-20592	Create a tool to verify tables do not have prefix tree encoding	Minor	Operability, tooling
HBASE-20046	Reconsider the implementation for serial replication	Major	Replication
HBASE-19397	Design procedures for ReplicationManager to notify peer change event from master	Major	proc-v2, Replication

IMPROVEMENTS:

JIRA	Summary	Priority	Component
HBASE-20806	Split style journal for flushes and compactions	Minor	.
HBASE-20474	Show non-RPC tasks on master/regionserver Web UI by default	Major	UI
HBASE-20826	Truncate responseInfo attributes on RpcServer WARN messages	Major	rpc
HBASE-20450	Provide metrics for number of total active, priority and replication rpc handlers	Major	metrics
HBASE-20810	Include the procedure id in the exception message in HBaseAdmin for better debugging	Major	Admin, proc-v2
HBASE-20040	Master UI should include “Cluster Key” needed to use the cluster as a replication sink	Minor	Replication, Usability
HBASE-20095	Redesign single instance pool in CleanerChore	Critical	.
HBASE-19164	Avoid UUID.randomUUID in tests	Major	test
HBASE-20739	Add priority for SCP	Major	Recovery
HBASE-20737	put collection into ArrayList instead of addAll function	Trivial	.
HBASE-20695	Implement table level RegionServer replication metrics	Minor	metrics
HBASE-20733	QABot should run checkstyle tests if the checkstyle configs change	Minor	build, community
HBASE-20625	refactor some WALCellCodec related code	Minor	wal
HBASE-19852	HBase Thrift 1 server SPNEGO Improvements	Major	Thrift
HBASE-20579	Improve snapshot manifest copy in ExportSnapshot	Minor	mapreduce
HBASE-20444	Improve version comparison logic for HBase specific version string and add unit tests	Major	util
HBASE-20594	provide utility to compare old and new descriptors	Major	.
HBASE-20640	TestQuotaGlobalsSettingsBypass missing test category and ClassRule	Critical	test
HBASE-20478	move import checks from hbaseanti to checkstyle	Minor	test
HBASE-20548	Master fails to startup on large clusters, refreshing block distribution	Major	.
HBASE-20488	PE tool prints full name in help message	Minor	shell
HBASE-20567	Pass both old and new descriptors to pre/post hooks of modify operations for table and namespace	Major	.
HBASE-20545	Improve performance of BaseLoadBalancer.retainAssignment	Major	Balancer
HBASE-16191	Add stop_regionserver and stop_master to shell	Major	.
HBASE-20536	Make TestRegionServerAccounting stable and it should not use absolute number	Minor	.
HBASE-20523	PE tool should support configuring client side buffering sizes	Minor	.
HBASE-20527	Remove unused code in MetaTableAccessor	Trivial	.
HBASE-20507	Do not need to call recoverLease on the broken file when we fail to create a wal writer	Major	wal
HBASE-20484	Remove the unnecessary autoboxing in FilterListBase	Trivial	.
HBASE-20327	When qualifier is not specified, append and incr operation do not work (shell)	Minor	shell
HBASE-20389	Move website building flags into a profile	Minor	build, website
HBASE-20379	shadedjars yetus plugin should add a footer link	Major	test
HBASE-20243	[Shell] Add shell command to create a new table by cloning the existent table	Minor	shell
HBASE-20286	Improving shell command compaction_state	Minor	shell
HBASE-19488	Move to using Apache commons CollectionUtils	Trivial	.
HBASE-20197	Review of ByteBufferWriterOutputStream.java	Minor	.
HBASE-20047	AuthenticationTokenIdentifier should provide a toString	Minor	Usability
HBASE-19024	Configurable default durability for synchronous WAL	Critical	wal
HBASE-19389	Limit concurrency of put with dense (hundreds) columns to prevent write handler exhausted	Critical	Performance
HBASE-20186	Improve RSGroupBasedLoadBalancer#balanceCluster() to be more efficient when calculating cluster state for each rsgroup	Minor	rsgroup
HBASE-19449	Minor logging change in HFileArchiver	Trivial	.
HBASE-20120	Remove some unused classes/ java files from hbase-server	Minor	.

BUG FIXES:

JIRA	Summary	Priority	Component
HBASE-20854	Wrong retries number in RpcRetryingCaller's log message	Minor	Client, logging
HBASE-20784	Will lose the SNAPSHOT suffix if we get the version of RS from ServerManager	Minor	master, UI
HBASE-20822	TestAsyncNonMetaRegionLocator is flakey	Major	asyncclient
HBASE-20808	Wrong shutdown order between Chores and ChoreService	Minor	.
HBASE-20789	TestBucketCache#testCacheBlockNextBlockMetadataMissing is flaky	Major	.
HBASE-20829	Remove the addFront assertion in MasterProcedureScheduler.doAdd	Major	Replication
HBASE-20825	Fix pre and post hooks of CloneSnapshot and RestoreSnapshot for Access checks	Major	security
HBASE-20812	Add defaults to Table Interface so implementors don't have to	Major	.
HBASE-20817	Infinite loop when executing ReopenTableRegionsProcedure	Blocker	Region Assignment
HBASE-20792	info:servername and info:sn inconsistent for OPEN region	Blocker	Region Assignment
HBASE-20769	getSplits() has a out of bounds problem in TableSnapshotInputFormatImpl	Major	.
HBASE-20732	Shutdown scan pool when master is stopped.	Minor	.
HBASE-20785	NPE getting metrics in PE testing scans	Major	Performance
HBASE-20795	Allow option in BBKVComparator.compare to do comparison without sequence id	Major	.
HBASE-20777	RpcConnection could still remain opened after we shutdown the NettyRpcServer	Major	rpc
HBASE-20403	Prefetch sometimes doesn't work with encrypted file system	Major	.
HBASE-20635	Support to convert the shaded user permission proto to client user permission object	Major	.
HBASE-20778	Make it so WALPE runs on DFS	Major	test
HBASE-20775	TestMultiParallel is flakey	Major	Region Assignment
HBASE-20752	Make sure the regions are truly reopened after ReopenTableRegionsProcedure	Major	proc-v2
HBASE-18622	Mitigate API compatibility concerns between branch-1 and branch-2	Blocker	API
HBASE-20767	Always close hbaseAdmin along with connection in HBTU	Major	test
HBASE-20642	IntegrationTestDDLMasterFailover throws 'InvalidFamilyOperationException	Major	.
HBASE-20742	Always create WAL directory for region server	Major	wal
HBASE-20708	Remove the usage of RecoverMetaProcedure in master startup	Blocker	proc-v2, Region Assignment
HBASE-20723	Custom hbase.wal.dir results in data loss because we write recovered edits into a different place than where the recovering region server looks for them	Critical	Recovery, wal
HBASE-20681	IntegrationTestDriver fails after HADOOP-15406 due to missing hamcrest-core	Major	integration tests
HBASE-20561	The way we stop a ReplicationSource may cause the RS down	Major	Replication
HBASE-19377	Compatibility checker complaining about hash collisions	Major	community
HBASE-20689	Docker fails to install rubocop for precommit	Blocker	build
HBASE-20707	Move MissingSwitchDefault check from checkstyle to error-prone	Major	build
HBASE-20699	QuotaCache should cancel the QuotaRefresherChore service inside its stop()	Major	.
HBASE-20590	REST Java client is not able to negotiate with the server in the secure mode	Critical	REST, security
HBASE-20683	Incorrect return value for PreUpgradeValidator	Critical	.
HBASE-20684	org.apache.hadoop.hbase.client.Scan#setStopRow javadoc uses incorrect method	Trivial	Client, documentation
HBASE-20678	NPE in ReplicationSourceManager#NodeFailoverWorker	Minor	.
HBASE-20670	NPE in HMaster#isInMaintenanceMode	Minor	.
HBASE-20634	Reopen region while server crash can cause the procedure to be stuck	Critical	.
HBASE-12882	Log level for org.apache.hadoop.hbase package should be configurable	Major	.
HBASE-20668	Avoid permission change if ExportSnapshot's copy fails	Major	.
HBASE-18116	Replication source in-memory accounting should not include bulk transfer hfiles	Major	Replication
HBASE-20602	hbase.master.quota.observer.ignore property seems to be not taking effect	Minor	documentation
HBASE-20664	Variable shared across multiple threads	Major	.
HBASE-20659	Implement a reopen table regions procedure	Major	.
HBASE-20582	Bump up JRuby version because of some reported vulnerabilities	Major	dependencies, shell
HBASE-20533	Fix the flaky TestAssignmentManagerMetrics	Major	.
HBASE-20597	Use a lock to serialize access to a shared reference to ZooKeeperWatcher in HBaseReplicationEndpoint	Minor	Replication
HBASE-20633	Dropping a table containing a disable violation policy fails to remove the quota upon table delete	Major	.
HBASE-20645	Fix security_available method in security.rb	Major	.
HBASE-20612	TestReplicationKillSlaveRSWithSeparateOldWALs sometimes fail because it uses an expired cluster conn	Major	.
HBASE-20648	HBASE-19364 “Truncate_preserve fails with table when replica region > 1” for master branch	Major	.
HBASE-20588	Space quota change after quota violation doesn't seem to take in effect	Major	regionserver
HBASE-20616	TruncateTableProcedure is stuck in retry loop in TRUNCATE_TABLE_CREATE_FS_LAYOUT state	Major	amv2
HBASE-20638	nightly source artifact testing should fail the stage if it's going to report an error on jira	Major	test
HBASE-20624	Race in ReplicationSource which causes walEntryFilter being null when creating new shipper	Major	Replication
HBASE-20601	Add multiPut support and other miscellaneous to PE	Minor	tooling
HBASE-20627	Relocate RS Group pre/post hooks from RSGroupAdminServer to RSGroupAdminEndpoint	Major	.
HBASE-20591	nightly job doesn't respect maven options	Critical	test
HBASE-20560	Revisit the TestReplicationDroppedTables ut	Major	.
HBASE-20571	JMXJsonServlet generates invalid JSON if it has NaN in metrics	Major	UI
HBASE-20585	Need to clear peer map when clearing MasterProcedureScheduler	Major	proc-v2
HBASE-20457	Return immediately for a scan rpc call when we want to switch from pread to stream	Major	scan
HBASE-20447	Only fail cacheBlock if block collisions aren't related to next block metadata	Major	BlockCache, BucketCache
HBASE-20544	downstream HBaseTestingUtility fails with invalid port	Blocker	test
HBASE-20004	Client is not able to execute REST queries in a secure cluster	Minor	REST, security
HBASE-20475	Fix the flaky TestReplicationDroppedTables unit test.	Major	.
HBASE-20554	“WALs outstanding” message from CleanerChore is noisy	Trivial	.
HBASE-20204	Add locking to RefreshFileConnections in BucketCache	Major	BucketCache
HBASE-20485	Copy constructor of Scan doesn't copy the readType and replicaId	Minor	.
HBASE-20543	Fix the flaky TestThriftHttpServer	Major	.
HBASE-20521	TableOutputFormat.checkOutputSpecs conf checking sequence cause pig script run fail	Major	mapreduce
HBASE-20500	[rsgroup] should keep at least one server in default group	Major	rsgroup
HBASE-20517	Fix PerformanceEvaluation ‘column’ parameter	Major	test
HBASE-20524	Need to clear metrics when ReplicationSourceManager refresh replication sources	Minor	.
HBASE-20476	Open sequence number could go backwards in AssignProcedure	Major	Region Assignment
HBASE-20506	Add doc and test for unused RetryCounter, useful-looking utility	Minor	.
HBASE-20492	UnassignProcedure is stuck in retry loop on region stuck in OPENING state	Critical	amv2
HBASE-20497	The getRecoveredQueueStartPos always return 0 in RecoveredReplicationSourceShipper	Major	Replication
HBASE-18842	The hbase shell clone_snaphost command returns bad error message	Minor	shell
HBASE-20466	Consistently use override mechanism for exempt classes in CoprocessClassloader	Major	Coprocessors
HBASE-20006	TestRestoreSnapshotFromClientWithRegionReplicas is flakey	Critical	read replicas
HBASE-18059	The scanner order for memstore scanners are wrong	Critical	regionserver, scan, Scanners
HBASE-20404	Ugly cleanerchore complaint that dir is not empty	Major	master
HBASE-20419	Fix potential NPE in ZKUtil#listChildrenAndWatchForNewChildren callers	Major	.
HBASE-20364	nightly job gives old results or no results for stages that timeout on SCM	Critical	test
HBASE-20335	nightly jobs no longer contain machine information	Critical	test
HBASE-20338	WALProcedureStore#recoverLease() should have fixed sleeps for retrying rollWriter()	Major	.
HBASE-20356	make skipping protoc possible	Critical	dependencies, thirdparty
HBASE-15291	FileSystem not closed in secure bulkLoad	Major	.
HBASE-20068	Hadoopcheck project health check uses default maven repo instead of yetus managed ones	Major	community, test
HBASE-20361	Non-successive TableInputSplits may wrongly be merged by auto balancing feature	Major	mapreduce
HBASE-20260	Purge old content from the book for branch-2/master	Critical	documentation
HBASE-20058	improper quoting in presplitting command docs	Minor	documentation
HBASE-19923	Reset peer state and config when refresh replication source failed	Major	Replication
HBASE-19748	TestRegionReplicaFailover and TestRegionReplicaReplicationEndpoint UT hangs	Major	.

TESTS:

JIRA	Summary	Priority	Component
HBASE-20667	Rename TestGlobalThrottler to TestReplicationGlobalThrottler	Trivial	.
HBASE-20646	TestWALProcedureStoreOnHDFS failing on branch-1	Trivial	.
HBASE-20505	PE should support multi column family read and write cases	Minor	.
HBASE-20513	Collect and emit ScanMetrics in PerformanceEvaluation	Minor	test
HBASE-20414	TestLockProcedure#testMultipleLocks may fail on slow machine	Major	.

SUB-TASKS:

JIRA	Summary	Priority	Component
HBASE-20831	Copy master doc into branch-2.1 and edit to make it suit 2.1.0	Blocker	documentation
HBASE-20839	Fallback to FSHLog if we can not instantiated AsyncFSWAL when user does not specify AsyncFSWAL explicitly	Blocker	wal
HBASE-20244	NoSuchMethodException when retrieving private method decryptEncryptedDataEncryptionKey from DFSClient	Blocker	wal
HBASE-20193	Basic Replication Web UI - Regionserver	Critical	Replication, Usability
HBASE-20489	Update Reference Guide that CLUSTER_KEY value is present on the Master UI info page.	Minor	documentation
HBASE-19722	Meta query statistics metrics source	Major	.
HBASE-20781	Save recalculating families in a WALEdit batch of Cells	Major	Performance
HBASE-20194	Basic Replication WebUI - Master	Critical	Replication, Usability
HBASE-20780	ServerRpcConnection logging cleanup	Major	logging, Performance
HBASE-19764	Fix Checkstyle errors in hbase-endpoint	Minor	.
HBASE-20710	extra cloneFamily() in Mutation.add(Cell)	Minor	regionserver
HBASE-18569	Add prefetch support for async region locator	Major	asyncclient, Client
HBASE-20706	[hack] Don't add known not-OPEN regions in reopen phase of MTP	Critical	amv2
HBASE-20334	add a test that expressly uses both our shaded client and the one from hadoop 3	Major	hadoop3, shading
HBASE-20615	emphasize use of shaded client jars when they're present in an install	Major	build, Client, Usability
HBASE-20333	break up shaded client into one with no Hadoop and one that's standalone	Critical	shading
HBASE-20332	shaded mapreduce module shouldn't include hadoop	Critical	mapreduce, shading
HBASE-20722	Make RegionServerTracker only depend on children changed event	Major	.
HBASE-20700	Move meta region when server crash can cause the procedure to be stuck	Critical	master, proc-v2, Region Assignment
HBASE-20628	SegmentScanner does over-comparing when one flushing	Critical	Performance
HBASE-19761	Fix Checkstyle errors in hbase-zookeeper	Minor	.
HBASE-19724	Fix Checkstyle errors in hbase-hadoop2-compat	Minor	.
HBASE-20518	Need to serialize the enabled field for UpdatePeerConfigProcedure	Major	Replication
HBASE-20481	Replicate entries from same region serially in ReplicationEndpoint for serial replication	Major	.
HBASE-20378	Provide a hbck option to cleanup replication barrier for a table	Major	.
HBASE-20128	Add new UTs which extends the old replication UTs but set replication scope to SERIAL	Major	.
HBASE-20417	Do not read wal entries when peer is disabled	Major	Replication
HBASE-20294	Also cleanup last pushed sequence id in ReplicationBarrierCleaner	Major	Replication
HBASE-20377	Deal with table in enabling and disabling state when modifying serial replication peer	Major	Replication
HBASE-20367	Write a replication barrier for regions when disabling a table	Major	Replication
HBASE-20296	Remove last pushed sequence ids when removing tables from a peer	Major	Replication
HBASE-20285	Delete all last pushed sequence ids when removing a peer or removing the serial flag for a peer	Major	Replication
HBASE-20138	Find a way to deal with the conflicts when updating replication position	Major	Replication
HBASE-20127	Add UT for serial replication after failover	Major	Replication, test
HBASE-20271	ReplicationSourceWALReader.switched should use the file name instead of the path object directly	Major	Replication
HBASE-20227	Add UT for ReplicationUtils.contains method	Major	Replication, test
HBASE-20147	Serial replication will be stuck if we create a table with serial replication but add it to a peer after there are region moves	Major	.
HBASE-20116	Optimize the region last pushed sequence id layout on zk	Major	Replication
HBASE-20242	The open sequence number will grow if we fail to open a region after writing the max sequence id file	Major	.
HBASE-20155	update branch-2 version to 2.1.0-SNAPSHOT	Major	build, community
HBASE-20206	WALEntryStream should not switch WAL file silently	Major	Replication
HBASE-20117	Cleanup the unused replication barriers in meta table	Major	master, Replication
HBASE-20165	Shell command to make a normal peer to be a serial replication peer	Major	.
HBASE-20167	Optimize the implementation of ReplicationSourceWALReader	Major	Replication
HBASE-20125	Add UT for serial replication after region split and merge	Major	Replication
HBASE-20129	Add UT for serial replication checker	Major	Replication
HBASE-20115	Reimplement serial replication based on the new replication storage layer	Major	Replication
HBASE-20050	Reimplement updateReplicationPositions logic in serial replication based on the newly introduced replication storage layer	Major	.
HBASE-20082	Fix findbugs errors only on master which are introduced by HBASE-19397	Major	findbugs
HBASE-19936	Introduce a new base class for replication peer procedure	Major	.
HBASE-19719	Fix checkstyle issues	Major	proc-v2, Replication
HBASE-19711	TestReplicationAdmin.testConcurrentPeerOperations hangs	Major	proc-v2
HBASE-19707	Race in start and terminate of a replication source after we async start replicatione endpoint	Major	proc-v2, Replication
HBASE-19636	All rs should already start work with the new peer change when replication peer procedure is finished	Major	proc-v2, Replication
HBASE-19634	Add permission check for executeProcedures in AccessController	Major	proc-v2, Replication
HBASE-19697	Remove TestReplicationAdminUsingProcedure	Major	proc-v2, Replication
HBASE-19661	Replace ReplicationStateZKBase with ZKReplicationStorageBase	Major	proc-v2, Replication
HBASE-19687	Move the logic in ReplicationZKNodeCleaner to ReplicationChecker and remove ReplicationZKNodeCleanerChore	Major	proc-v2, Replication
HBASE-19544	Add UTs for testing concurrent modifications on replication peer	Major	proc-v2, Replication, test
HBASE-19686	Use KeyLocker instead of ReentrantLock in PeerProcedureHandlerImpl	Major	proc-v2, Replication
HBASE-19623	Create replication endpoint asynchronously when adding a replication source	Major	proc-v2, Replication
HBASE-19633	Clean up the replication queues in the postPeerModification stage when removing a peer	Major	proc-v2, Replication
HBASE-19622	Reimplement ReplicationPeers with the new replication storage interface	Major	proc-v2, Replication
HBASE-19635	Introduce a thread at RS side to call reportProcedureDone	Major	proc-v2
HBASE-19617	Remove ReplicationQueues, use ReplicationQueueStorage directly	Major	Replication
HBASE-19642	Fix locking for peer modification procedure	Critical	proc-v2, Replication
HBASE-19592	Add UTs to test retry on update zk failure	Major	proc-v2, Replication
HBASE-19630	Add peer cluster key check when add new replication peer	Major	proc-v2, Replication
HBASE-19573	Rewrite ReplicationPeer with the new replication storage interface	Major	proc-v2, Replication
HBASE-19579	Add peer lock test for shell command list_locks	Major	proc-v2, Replication
HBASE-19599	Remove ReplicationQueuesClient, use ReplicationQueueStorage directly	Major	Replication
HBASE-19543	Abstract a replication storage interface to extract the zk specific code	Major	proc-v2, Replication
HBASE-19525	RS side changes for moving peer modification from zk watcher to procedure	Major	proc-v2, Replication
HBASE-19580	Use slf4j instead of commons-logging in new, just-added Peer Procedure classes	Major	proc-v2, Replication
HBASE-19520	Add UTs for the new lock type PEER	Major	proc-v2
HBASE-19564	Procedure id is missing in the response of peer related operations	Major	proc-v2, Replication
HBASE-19536	Client side changes for moving peer modification from zk watcher to procedure	Major	Replication
HBASE-19524	Master side changes for moving peer modification from zk watcher to procedure	Major	proc-v2, Replication
HBASE-19216	Implement a general framework to execute remote procedure on RS	Major	proc-v2, Replication

OTHER:

JIRA	Summary	Priority	Component
HBASE-19997	[rolling upgrade] 1.x => 2.x	Blocker	.
HBASE-20360	Further optimization for serial replication	Major	Replication
HBASE-20862	Address 2.1.0 Compatibility Report Issues	Blocker	compatibility
HBASE-20665	“Already cached block XXX” message should be DEBUG	Minor	BlockCache
HBASE-20677	Backport test of HBASE-20566 ‘Creating a system table after enabling rsgroup feature puts region into RIT’ to branch-2	Major	.
HBASE-19475	Extend backporting strategy in documentation	Trivial	documentation
HBASE-20595	Remove the concept of ‘special tables’ from rsgroups	Major	Region Assignment, rsgroup
HBASE-20415	branches-2 don't need maven-scala-plugin	Major	build
HBASE-20112	Include test results from nightly hadoop3 tests in jenkins test results	Critical	test
HBASE-17918	document serial replication	Critical	documentation, Replication
HBASE-19737	Manage a HBASE-19397-branch-2 branch and merge it to branch-2	Major	proc-v2, Replication