HBASE Changelog

Release 2.1.2 - Unreleased (as of 2018-12-07)

IMPROVEMENTS:

JIRASummaryPriorityComponent
HBASE-21413Empty meta log doesn't get split when restart whole clusterMajor.
HBASE-21524Unnecessary DEBUG log in ConnectionImplementation#isTableEnabledMajorClient
HBASE-21511Remove in progress snapshot check in SnapshotFileCache#getUnreferencedFilesMinor.
HBASE-21480Taking snapshot when RS crashes prevent we bring the regions onlineMajorsnapshots
HBASE-21485Add more debug logs for remote procedure executionMajorproc-v2
HBASE-21388No need to instantiate MemStoreLAB for master which not carry tableMajor.

BUG FIXES:

JIRASummaryPriorityComponent
HBASE-21559The RestoreSnapshotFromClientTestBase related UT are flakyMajor.
HBASE-21551Memory leak when use scan with STREAM at server sideBlockerregionserver
HBASE-21479Individual tests in TestHRegionReplayEvents class are failingMajor.
HBASE-21518TestMasterFailoverWithProcedures is flakyMajor.
HBASE-21504If enable FIFOCompactionPolicy, a compaction may write a “empty” hfile whose maxTimeStamp is long max. This kind of hfile will never be archived.CriticalCompaction
HBASE-21300Fix the wrong reference file path when restoring snapshots for tables with MOB columnsMajor.
HBASE-21492CellCodec Written To WAL Before It's VerifiedCriticalwal
HBASE-21507Compaction failed when execute AbstractMultiFileWriter.beforeShipped() methodMajorregionserver
HBASE-21387Race condition surrounding in progress snapshot handling in snapshot cache leads to loss of snapshot filesMajorsnapshots
HBASE-21503Replication normal source can get stuck due potential race conditions between source wal reader and wal provider initialization threads.BlockerReplication
HBASE-21440Assign procedure on the crashed server is not properly interruptedMajor.
HBASE-21468separate workers for meta table is not workingMajor.
HBASE-21445CopyTable by bulkload will write hfile into yarn's HDFSMajormapreduce
HBASE-21437Bypassed procedure throw IllegalArgumentException when its state is WAITING_TIMEOUTMajor.
HBASE-21439StochasticLoadBalancer RegionLoads aren’t being used in RegionLoad cost functionsMajorBalancer
HBASE-20604ProtobufLogReader#readNext can incorrectly loop to the same position in the stream until the the WAL is rolledCriticalReplication, wal
HBASE-21247Custom Meta WAL Provider doesn't default to custom WAL Provider whose configuration value is outside the enums in ProvidersMajorwal
HBASE-21438TestAdmin2#testGetProcedures fails due to FailedProcedure inaccessibleMajor.
HBASE-214252.1.1 fails to start over 1.x data; namespace not assignedCriticalamv2
HBASE-21407Resolve NPE in backup Master UIMinorUI
HBASE-21424Change flakies and nightlies so scheduled less oftenMajorbuild
HBASE-21417Pre commit build is broken due to surefire plugin crashesCriticalbuild
HBASE-21371Hbase unable to compile against Hadoop trunk (3.3.0-SNAPSHOT) due to license errorMajor.
HBASE-21356bulkLoadHFile API should ensure that rs has the source hfile's write permissionMajor.
HBASE-21055NullPointerException when balanceOverall() but server balance info is nullMajorBalancer

SUB-TASKS:

JIRASummaryPriorityComponent
HBASE-21490WALProcedure may remove proc wal files still with active proceduresMajorproc-v2
HBASE-21377Add debug log for procedure stack id related operationsMajorproc-v2
HBASE-21473RowIndexSeekerV1 may return cell with extra two \x00\x00 bytes which has no tagsMajor.
HBASE-21423Procedures for meta table/region should be able to execute in separate workersMajor.
HBASE-21376Add some verbose log to MasterProcedureSchedulerMajorlogging, proc-v2
HBASE-21442Update branch-2.1 for next development cycleMajorbuild
HBASE-21421Do not kill RS if reportOnlineRegions failsMajor.
HBASE-21314The implementation of BitSetNode is not efficientMajorproc-v2
HBASE-21395Abort split/merge procedure if there is a table procedure of the same table going onMajor.
HBASE-21351The force update thread may have race with PE worker when the procedure is rolling backCriticalproc-v2
HBASE-21237Use CompatRemoteProcedureResolver to dispatch open/close region requests to RSBlocker.
HBASE-21322Add a scheduleServerCrashProcedure() API to HbckServiceMajorhbck2
HBASE-21375Revisit the lock and queue implementation in MasterProcedureSchedulerMajorproc-v2

OTHER:

JIRASummaryPriorityComponent
HBASE-18735Provide a fast mechanism for shutting down mini clusterMajor.
HBASE-21517Move the getTableRegionForRow method from HMaster to TestMasterMajortest

Release 2.1.1 - Released 2018-10-31

INCOMPATIBLE CHANGES:

JIRASummaryPriorityComponent
HBASE-21158Empty qualifier cell should not be returned if it does not match QualifierFilterCriticalFilters
HBASE-21223[amv2] Remove abort_procedure from shellCriticalamv2, hbck2, shell
HBASE-20884Replace usage of our Base64 implementation with java.util.Base64Major.

NEW FEATURES:

JIRASummaryPriorityComponent
HBASE-20649Validate HFiles do not have PREFIX_TREE DataBlockEncodingMinorOperability, tooling

IMPROVEMENTS:

JIRASummaryPriorityComponent
HBASE-21385HTable.delete request use rpc call directly instead of AsyncProcessMajor.
HBASE-21145Backport “HBASE-21126 Add ability for HBase Canary to ignore a configurable number of ZooKeeper down nodes” to branch-2.1Minorcanary, Zookeeper
HBASE-21263Mention compression algorithm along with other storefile detailsMinor.
HBASE-21290No need to instantiate BlockCache for master which not carry tableMajor.
HBASE-21251Refactor RegionMoverMajorOperability
HBASE-21303[shell] clear_deadservers with no args failsMajor.
HBASE-21299List counts of actual region states in master UI tables sectionMajorUI
HBASE-21289Remove the log “‘hbase.regionserver.maxlogs’ was deprecated.” in AbstractFSWALMinor.
HBASE-21185WALPrettyPrinter: Additional useful info to be printed by wal printer tool, for debugability purposesMinorOperability
HBASE-21103nightly test cache of yetus install needs to be more thorough in verificationMajortest
HBASE-20857JMX - add Balancer status = enabled / disabledMajorAPI, master, metrics, REST, tooling, Usability
HBASE-21164reportForDuty to spew less log if master is initializingMinorregionserver
HBASE-20307LoadTestTool prints too much zookeeper loggingMinortooling
HBASE-21155Save on a few log strings and some churn in wal splitter by skipping out early if no logs in dirTrivial.
HBASE-21157Split TableInputFormatScan to individual testsMinortest
HBASE-21153Shaded client jars should always build in relevant phase to avoid confusionMajorbuild
HBASE-20749Upgrade our use of checkstyle to 8.6+Minorbuild, community
HBASE-20387flaky infrastructure should work for all branchesCriticaltest
HBASE-20469Directory used for sidelining old recovered edits files should be made configurableMinor.
HBASE-20979Flaky test reporting should specify what JSON it needs and handle HTTP errorsMinortest
HBASE-20985add two attributes when we do normalizationMajor.
HBASE-20965Separate region server report requests to new handlersMajorPerformance
HBASE-20986Separate the config of block size when we do log splitting and write HlogMajor.
HBASE-20856PITA having to set WAL provider in two placesMinorOperability, wal
HBASE-20935HStore.removeCompactedFiles should log in case it is unable to delete a fileMinor.
HBASE-20873Update doc for Endpoint-based ExportMinordocumentation

BUG FIXES:

JIRASummaryPriorityComponent
HBASE-21380TestRSGroups failingMajor.
HBASE-21391RefreshPeerProcedure should also wait master initialized before executingMajorReplication
HBASE-21342FileSystem in use may get closed by other bulk load call in secure bulkLoadMajor.
HBASE-21349Cluster is going down but CatalogJanitor and Normalizer try to run and fail noiselyMinor.
HBASE-21355HStore's storeSize is calculated repeatedly which causing the confusing region splitBlockerregionserver
HBASE-21334TestMergeTableRegionsProcedure is flakeyMajoramv2, proc-v2, test
HBASE-21178[BC break] : Get and Scan operation with a custom converter_class not workingCriticalshell
HBASE-21242[amv2] Miscellaneous minor log and assign procedure create improvementsMinoramv2, Operability
HBASE-21348Fix failing TestRegionBypass, broke by HBASE-21291Majorhbck2
HBASE-21345[hbck2] Allow version check to proceed even though master is ‘initializing’.Majorhbck2
HBASE-21200Memstore flush doesn't finish because of seekToPreviousRow() in memstore scanner.CriticalScanners
HBASE-21292IdLock.getLockEntry() may hang if interruptedMajor.
HBASE-21335Change the default wait time of HBCK2 toolCritical.
HBASE-21291Add a test for bypassing stuck state-machine proceduresMajor.
HBASE-21327Fix minor logging issue where we don't report servername if no associated SCPTrivialamv2
HBASE-21320[canary] Cleanup of usage and add commentaryMajorcanary
HBASE-21266Not running balancer because processing dead regionservers, but empty dead rs listMajor.
HBASE-21260The whole balancer plans might be aborted if there are more than one plans to move a same regionMajorBalancer, master
HBASE-21271[amv2] Don't throw UnsupportedOperationException when rollback called on Assign/Unassign; spiral of deathMajoramv2
HBASE-21259[amv2] Revived deadservers; recreated serverstatenodeCriticalamv2
HBASE-21280Add anchors for each heading in UITrivialUI, Usability
HBASE-20764build broken when latest commit is gpg signedCriticalbuild
HBASE-21213[hbck2] bypass leaves behind state in RegionStates when assign/unassignMajoramv2, hbck2
HBASE-18549Unclaimed replication queues can go undetectedCriticalReplication
HBASE-21248Implement exponential backoff when retrying for ModifyPeerProcedureMajorproc-v2, Replication
HBASE-21196HTableMultiplexer clears the meta cache after every put operationCriticalPerformance
HBASE-19418RANGE_OF_DELAY in PeriodicMemstoreFlusher should be configurable.Minor.
HBASE-18451PeriodicMemstoreFlusher should inspect the queue before adding a delayed flush requestMajorregionserver
HBASE-21228Memory leak since AbstractFSWAL caches Thread object and never clean laterCriticalwal
HBASE-21232Show table state in Tables view on Master home pageMajorOperability, UI
HBASE-21212Wrong flush time when update flush metricMinor.
HBASE-21208Bytes#toShort doesn't work without unsafeCritical.
HBASE-20704Sometimes some compacted storefiles are not archived on region closeCriticalCompaction
HBASE-21203TestZKMainServer#testCommandLineWorks won't pass with default 4lw whitelistMinortest, Zookeeper
HBASE-21206Scan with batch size may return incomplete cellsCriticalscan
HBASE-21182Failed to execute start-hbase.shMajor.
HBASE-21179Fix the number of actions in responseTooSlow logMajorlogging, rpc
HBASE-21174[REST] Failed to parse empty qualifier in TableResource#getScanResourceMajorREST
HBASE-21181Use the same filesystem for wal archive directory and wal directoryMajor.
HBASE-21144AssignmentManager.waitForAssignment is not stableMajoramv2, test
HBASE-21143Update findbugs-maven-plugin to 3.0.4Majorpom
HBASE-21171[amv2] Tool to parse a directory of MasterProcWALs standaloneMajoramv2, test
HBASE-21001ReplicationObserver fails to load in HBase 2.0.0Major.
HBASE-20741Split of a region with replicas creates all daughter regions and its replica in same serverMajorread replicas
HBASE-21127TableRecordReader need to handle cursor result tooMajor.
HBASE-20892[UI] Start / End keys are empty on table.jspMajor.
HBASE-21132return wrong result in rest multigetMajor.
HBASE-20940HStore.cansplit should not allow split to happen if it has referencesMajor.
HBASE-20968list_procedures_test fails due to no matching regexMajorshell, test
HBASE-21030Correct javadoc for append operationMinordocumentation
HBASE-21088HStoreFile should be closed in HStore#hasReferencesMajor.
HBASE-21120MoveRegionProcedure makes no progress; goes to STUCKMajoramv2
HBASE-20772Controlled shutdown fills Master log with the disturbing message "No matching procedure found for rit=OPEN, location=ZZZZ, table=YYYYY, region=XXXX transition to CLOSEDMajorlogging
HBASE-20978[amv2] Worker terminating UNNATURALLY during MoveRegionProcedureCriticalamv2
HBASE-21078[amv2] CODE-BUG NPE in RTP doing UnassignMajoramv2
HBASE-21113Apply the branch-2 version of HBASE-21095, The timeout retry logic for several procedures are broken after master restartsMajoramv2
HBASE-21101Remove the waitUntilAllRegionsAssigned call after split in TestTruncateTableProcedureMajortest
HBASE-20614REST scan API with incorrect filter text file throws HTTP 503 Service Unavailable errorMinorREST
HBASE-21041Memstore's heap size will be decreased to minus zero after flushMajor.
HBASE-21031Memory leak if replay edits failed during region openingMajor.
HBASE-21032ScanResponses contain only one cell eachMajorPerformance, Scanners
HBASE-20705Having RPC Quota on a table prevents Space quota to be recreated/removedMajor.
HBASE-21058Nightly tests for branches 1 fail to build ref guideMajordocumentation
HBASE-21074JDK7 branches need to pass “-Dhttps.protocols=TLSv1.2” to maven when buildingMajorbuild, community, test
HBASE-21062WALFactory has misleading notion of “default”Majorwal
HBASE-21047Object creation of StoreFileScanner thru constructor and close may leave refCount to -1Major.
HBASE-21005Maven site configuration causes downstream projects to get a directory named ${project.basedir}Minorbuild
HBASE-21029Miscount of memstore's heap/offheap size if same cell was putMajor.
HBASE-20981Rollback stateCount accounting thrown-off when exception out of rollbackStateMajoramv2
HBASE-21018RS crashed because AsyncFS was unable to update HDFS data encryption keyCriticalwal
HBASE-21007Memory leak in HBase rest serverCriticalREST
HBASE-20538Upgrade our hadoop versions to 2.7.7 and 3.0.3Criticaljava, security
HBASE-20565ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4MajorFilters
HBASE-20870Wrong HBase root dir in ITBLL's Search ToolMinorintegration tests
HBASE-20869Endpoint-based Export use incorrect user to write to destinationMajorCoprocessors
HBASE-20865CreateTableProcedure is stuck in retry loop in CREATE_TABLE_WRITE_FS_LAYOUT stateMajoramv2
HBASE-19572RegionMover should use the configured default port number and not the one from HConstantsMajor.
HBASE-20697Can't cache All region locations of the specify table by calling table.getRegionLocator().getAllRegionLocations()Majormeta

TESTS:

JIRASummaryPriorityComponent
HBASE-21261Add log4j.properties for hbase-rsgroup testsTrivial.
HBASE-21097Flush pressure assertion may fail in testFlushThroughputTuningMajorregionserver
HBASE-21161Enable the test added in HBASE-20741 that was removed accidentallyMinor.
HBASE-21076TestTableResource fails with NPEMajorREST, test
HBASE-20907Fix Intermittent failure on TestProcedurePriorityMajor.
HBASE-20838Include hbase-server in precommit test if CommonFSUtils is changedMajor.

SUB-TASKS:

JIRASummaryPriorityComponent
HBASE-21398Copy down docs, amend to suite branch-2.1, and then commitMajor.
HBASE-21397Set version to 2.1.1 on branch-2.1 in prep for first RCMajor.
HBASE-20973ArrayIndexOutOfBoundsException when rolling back procedureCriticalamv2
HBASE-21384Procedure with holdlock=false should not be restored lock when restartsBlocker.
HBASE-21364Procedure holds the lock should put to front of the queue after restartBlocker.
HBASE-21215Figure how to invoke hbck2; make it easy to findMajoramv2, hbck2
HBASE-21372Set hbase.assignment.maximum.attempts to Long.MAXMajoramv2
HBASE-21363Rewrite the buildingHoldCleanupTracker method in WALProcedureStoreMajorproc-v2
HBASE-21338[balancer] If balancer is an ill-fit for cluster size, it gives little indicationMajorBalancer, Operability
HBASE-21192Add HOW-TO repair damaged AMv2.Majoramv2
HBASE-21073“Maintenance mode” masterMajoramv2, hbck2, master
HBASE-21354Procedure may be deleted improperly during master restarts resulting in ‘Corrupt’Major.
HBASE-21321Backport HBASE-21278 to branch-2.1 and branch-2.0Critical.
HBASE-21336Simplify the implementation of WALProcedureMapMajorproc-v2
HBASE-21323Should not skip force updating for a sub procedure even if it has been finishedMajorproc-v2
HBASE-21075Confirm that we can (rolling) upgrade from 2.0.x and 2.1.x to 2.2.x after HBASE-20881Blockeramv2, proc-v2
HBASE-21288HostingServer in UnassignProcedure is not accurateMajoramv2, Balancer
HBASE-20716Unsafe access cleanupCriticalPerformance
HBASE-21310Split TestCloneSnapshotFromClientMajortest
HBASE-21311Split TestRestoreSnapshotFromClientMajortest
HBASE-21315The getActiveMinProcId and getActiveMaxProcId of BitSetNode are incorrect if there are no active procedureMajor.
HBASE-21309Increase the waiting timeout for TestProcedurePriorityMajortest
HBASE-21254Need to find a way to limit the number of proc wal filesCriticalproc-v2
HBASE-21250Refactor WALProcedureStore and add more comments for better understanding the implementationMajorproc-v2
HBASE-19275TestSnapshotFileCache never worked properlyMajor.
HBASE-21249Add jitter for ProcedureUtil.getBackoffTimeMsMajorproc-v2
HBASE-21233Allow the procedure implementation to skip persistence of the state after a executionMajorPerformance, proc-v2
HBASE-21214[hbck2] setTableState just sets hbase:meta state, not in-memory stateMajoramv2, hbck2
HBASE-21210Add bypassProcedure() API to HBCK2Majorhbase-operator-tools, hbck2
HBASE-21023Add bypassProcedureToCompletion() API to HbckServiceMajorhbck2
HBASE-21156[hbck2] Queue an assign of hbase:meta and bulk assign/unassignCriticalhbck2
HBASE-21169Initiate hbck2 tool in hbase-operator-tools repoMajorhbck2
HBASE-21191Add a holding-pattern if no assign for meta or namespace (Can happen if masterprocwals have been cleared).Majoramv2
HBASE-21172Reimplement the retry backoff logic for ReopenTableRegionsProcedureMajoramv2, proc-v2
HBASE-21189flaky job should gather machine statsMinortest
HBASE-21190Log files and count of entries in each as we load from the MasterProcWAL storeMajoramv2
HBASE-21083Introduce a mechanism to bypass the execution of a stuck procedureMajoramv2
HBASE-20941Create and implement HbckService in masterMajor.
HBASE-21072Block out HBCK1 in hbase2Majorhbck
HBASE-21093Move TestCreateTableProcedure.testMRegions to a separated fileMajortest
HBASE-21094Remove the explicit timeout config for TestTruncateTableProcedureMajortest
HBASE-21050Exclusive lock may be held by a SUCCESS state procedure foreverMajoramv2
HBASE-21044Disable flakey TestShell list_proceduresMajortest
HBASE-20975Lock may not be taken or released while rolling back procedureMajoramv2
HBASE-21025Add cache for TableStateManagerMajor.
HBASE-21012Revert the change of serializing TimeRangeTrackerCritical.
HBASE-20813Remove RPC quotas when the associated table/Namespace is dropped offMinor.
HBASE-20885Remove entry for RPC quota from hbase:quota when RPC quota is removed.Minor.
HBASE-20893Data loss if splitting region while ServerCrashProcedure executingMajor.
HBASE-19369HBase Should use Builder Pattern to Create Log Files while using WAL on Erasure CodingMajor.
HBASE-20939There will be race when we call suspendIfNotReady and then throw ProcedureSuspendedExceptionCriticalamv2
HBASE-20921Possible NPE in ReopenTableRegionsProcedureMajoramv2
HBASE-20938Set version to 2.1.1-SNAPSHOT for branch-2.1Majorbuild
HBASE-20867RS may get killed while master restartsMajor.
HBASE-20878Data loss if merging regions while ServerCrashProcedure executingCriticalamv2
HBASE-20846Restore procedure locks when master restartsMajor.
HBASE-20914Trim Master memory usageMajormaster
HBASE-20853Polish “Add defaults to Table Interface so Implementors don't have to”MajorAPI
HBASE-20875MemStoreLABImp::copyIntoCell uses 7% CPU when writingMajorPerformance
HBASE-20860Merged region's RIT state may not be cleaned after master restartMajor.
HBASE-20847The parent procedure of RegionTransitionProcedure may not have the table lockMajorproc-v2, Region Assignment

OTHER:

JIRASummaryPriorityComponent
HBASE-20152[AMv2] DisableTableProcedure versus ServerCrashProcedureMajoramv2
HBASE-20540[umbrella] Hadoop 3 compatibilityMajor.
HBASE-21294[2.1] Update bouncycastle dependency.Majordependencies, test
HBASE-21198Exclude dependency on net.minidev:json-smartMajor.
HBASE-21114website should have a copy of 2.1 release docsMajorcommunity, documentation
HBASE-21287JVMClusterUtil Master initialization wait time not configurableMajortest
HBASE-21168BloomFilterUtil uses hardcoded randomnessTrivial.
HBASE-21125Backport ‘HBASE-20942 Improve RpcServer TRACE logging’ to branch-2.1MajorOperability
HBASE-20482Print a link to the ref guide chapter for the shell during startupMinordocumentation, shell

Release 2.1.0 - Released 2018-07-22

INCOMPATIBLE CHANGES:

JIRASummaryPriorityComponent
HBASE-20691Storage policy should allow deferring to HDFSBlockerFilesystem Integration, wal
HBASE-20270Turn off command help that follows all errors in shellMajorshell
HBASE-20501Change the Hadoop minimum version to 2.7.1Blockercommunity, documentation
HBASE-20406HBase Thrift HTTP - Shouldn't handle TRACE/OPTIONS methodsMajorsecurity, Thrift
HBASE-20159Support using separate ZK quorums for clientMajorClient, Operability, Zookeeper
HBASE-20148Make serial replication as a option for a peer instead of a tableMajorReplication

NEW FEATURES:

JIRASummaryPriorityComponent
HBASE-20833Modify pre-upgrade coprocessor validator to support table level coprocessorsMajorCoprocessors
HBASE-15809Basic Replication WebUICriticalReplication, UI
HBASE-19735Create a minimal “client” tarball installationMajorbuild, Client
HBASE-20656Validate pre-2.0 coprocessors against HBase 2.0+Majortooling
HBASE-20592Create a tool to verify tables do not have prefix tree encodingMinorOperability, tooling
HBASE-20046Reconsider the implementation for serial replicationMajorReplication
HBASE-19397Design procedures for ReplicationManager to notify peer change event from masterMajorproc-v2, Replication

IMPROVEMENTS:

JIRASummaryPriorityComponent
HBASE-20806Split style journal for flushes and compactionsMinor.
HBASE-20474Show non-RPC tasks on master/regionserver Web UI by defaultMajorUI
HBASE-20826Truncate responseInfo attributes on RpcServer WARN messagesMajorrpc
HBASE-20450Provide metrics for number of total active, priority and replication rpc handlersMajormetrics
HBASE-20810Include the procedure id in the exception message in HBaseAdmin for better debuggingMajorAdmin, proc-v2
HBASE-20040Master UI should include “Cluster Key” needed to use the cluster as a replication sinkMinorReplication, Usability
HBASE-20095Redesign single instance pool in CleanerChoreCritical.
HBASE-19164Avoid UUID.randomUUID in testsMajortest
HBASE-20739Add priority for SCPMajorRecovery
HBASE-20737put collection into ArrayList instead of addAll functionTrivial.
HBASE-20695Implement table level RegionServer replication metricsMinormetrics
HBASE-20733QABot should run checkstyle tests if the checkstyle configs changeMinorbuild, community
HBASE-20625refactor some WALCellCodec related codeMinorwal
HBASE-19852HBase Thrift 1 server SPNEGO ImprovementsMajorThrift
HBASE-20579Improve snapshot manifest copy in ExportSnapshotMinormapreduce
HBASE-20444Improve version comparison logic for HBase specific version string and add unit testsMajorutil
HBASE-20594provide utility to compare old and new descriptorsMajor.
HBASE-20640TestQuotaGlobalsSettingsBypass missing test category and ClassRuleCriticaltest
HBASE-20478move import checks from hbaseanti to checkstyleMinortest
HBASE-20548Master fails to startup on large clusters, refreshing block distributionMajor.
HBASE-20488PE tool prints full name in help messageMinorshell
HBASE-20567Pass both old and new descriptors to pre/post hooks of modify operations for table and namespaceMajor.
HBASE-20545Improve performance of BaseLoadBalancer.retainAssignmentMajorBalancer
HBASE-16191Add stop_regionserver and stop_master to shellMajor.
HBASE-20536Make TestRegionServerAccounting stable and it should not use absolute numberMinor.
HBASE-20523PE tool should support configuring client side buffering sizesMinor.
HBASE-20527Remove unused code in MetaTableAccessorTrivial.
HBASE-20507Do not need to call recoverLease on the broken file when we fail to create a wal writerMajorwal
HBASE-20484Remove the unnecessary autoboxing in FilterListBaseTrivial.
HBASE-20327When qualifier is not specified, append and incr operation do not work (shell)Minorshell
HBASE-20389Move website building flags into a profileMinorbuild, website
HBASE-20379shadedjars yetus plugin should add a footer linkMajortest
HBASE-20243[Shell] Add shell command to create a new table by cloning the existent tableMinorshell
HBASE-20286Improving shell command compaction_stateMinorshell
HBASE-19488Move to using Apache commons CollectionUtilsTrivial.
HBASE-20197Review of ByteBufferWriterOutputStream.javaMinor.
HBASE-20047AuthenticationTokenIdentifier should provide a toStringMinorUsability
HBASE-19024Configurable default durability for synchronous WALCriticalwal
HBASE-19389Limit concurrency of put with dense (hundreds) columns to prevent write handler exhaustedCriticalPerformance
HBASE-20186Improve RSGroupBasedLoadBalancer#balanceCluster() to be more efficient when calculating cluster state for each rsgroupMinorrsgroup
HBASE-19449Minor logging change in HFileArchiverTrivial.
HBASE-20120Remove some unused classes/ java files from hbase-serverMinor.

BUG FIXES:

JIRASummaryPriorityComponent
HBASE-20854Wrong retries number in RpcRetryingCaller's log messageMinorClient, logging
HBASE-20784Will lose the SNAPSHOT suffix if we get the version of RS from ServerManagerMinormaster, UI
HBASE-20822TestAsyncNonMetaRegionLocator is flakeyMajorasyncclient
HBASE-20808Wrong shutdown order between Chores and ChoreServiceMinor.
HBASE-20789TestBucketCache#testCacheBlockNextBlockMetadataMissing is flakyMajor.
HBASE-20829Remove the addFront assertion in MasterProcedureScheduler.doAddMajorReplication
HBASE-20825Fix pre and post hooks of CloneSnapshot and RestoreSnapshot for Access checksMajorsecurity
HBASE-20812Add defaults to Table Interface so implementors don't have toMajor.
HBASE-20817Infinite loop when executing ReopenTableRegionsProcedureBlockerRegion Assignment
HBASE-20792info:servername and info:sn inconsistent for OPEN regionBlockerRegion Assignment
HBASE-20769getSplits() has a out of bounds problem in TableSnapshotInputFormatImplMajor.
HBASE-20732Shutdown scan pool when master is stopped.Minor.
HBASE-20785NPE getting metrics in PE testing scansMajorPerformance
HBASE-20795Allow option in BBKVComparator.compare to do comparison without sequence idMajor.
HBASE-20777RpcConnection could still remain opened after we shutdown the NettyRpcServerMajorrpc
HBASE-20403Prefetch sometimes doesn't work with encrypted file systemMajor.
HBASE-20635Support to convert the shaded user permission proto to client user permission objectMajor.
HBASE-20778Make it so WALPE runs on DFSMajortest
HBASE-20775TestMultiParallel is flakeyMajorRegion Assignment
HBASE-20752Make sure the regions are truly reopened after ReopenTableRegionsProcedureMajorproc-v2
HBASE-18622Mitigate API compatibility concerns between branch-1 and branch-2BlockerAPI
HBASE-20767Always close hbaseAdmin along with connection in HBTUMajortest
HBASE-20642IntegrationTestDDLMasterFailover throws 'InvalidFamilyOperationExceptionMajor.
HBASE-20742Always create WAL directory for region serverMajorwal
HBASE-20708Remove the usage of RecoverMetaProcedure in master startupBlockerproc-v2, Region Assignment
HBASE-20723Custom hbase.wal.dir results in data loss because we write recovered edits into a different place than where the recovering region server looks for themCriticalRecovery, wal
HBASE-20681IntegrationTestDriver fails after HADOOP-15406 due to missing hamcrest-coreMajorintegration tests
HBASE-20561The way we stop a ReplicationSource may cause the RS downMajorReplication
HBASE-19377Compatibility checker complaining about hash collisionsMajorcommunity
HBASE-20689Docker fails to install rubocop for precommitBlockerbuild
HBASE-20707Move MissingSwitchDefault check from checkstyle to error-proneMajorbuild
HBASE-20699QuotaCache should cancel the QuotaRefresherChore service inside its stop()Major.
HBASE-20590REST Java client is not able to negotiate with the server in the secure modeCriticalREST, security
HBASE-20683Incorrect return value for PreUpgradeValidatorCritical.
HBASE-20684org.apache.hadoop.hbase.client.Scan#setStopRow javadoc uses incorrect methodTrivialClient, documentation
HBASE-20678NPE in ReplicationSourceManager#NodeFailoverWorkerMinor.
HBASE-20670NPE in HMaster#isInMaintenanceModeMinor.
HBASE-20634Reopen region while server crash can cause the procedure to be stuckCritical.
HBASE-12882Log level for org.apache.hadoop.hbase package should be configurableMajor.
HBASE-20668Avoid permission change if ExportSnapshot's copy failsMajor.
HBASE-18116Replication source in-memory accounting should not include bulk transfer hfilesMajorReplication
HBASE-20602hbase.master.quota.observer.ignore property seems to be not taking effectMinordocumentation
HBASE-20664Variable shared across multiple threadsMajor.
HBASE-20659Implement a reopen table regions procedureMajor.
HBASE-20582Bump up JRuby version because of some reported vulnerabilitiesMajordependencies, shell
HBASE-20533Fix the flaky TestAssignmentManagerMetricsMajor.
HBASE-20597Use a lock to serialize access to a shared reference to ZooKeeperWatcher in HBaseReplicationEndpointMinorReplication
HBASE-20633Dropping a table containing a disable violation policy fails to remove the quota upon table deleteMajor.
HBASE-20645Fix security_available method in security.rbMajor.
HBASE-20612TestReplicationKillSlaveRSWithSeparateOldWALs sometimes fail because it uses an expired cluster connMajor.
HBASE-20648HBASE-19364 “Truncate_preserve fails with table when replica region > 1” for master branchMajor.
HBASE-20588Space quota change after quota violation doesn't seem to take in effectMajorregionserver
HBASE-20616TruncateTableProcedure is stuck in retry loop in TRUNCATE_TABLE_CREATE_FS_LAYOUT stateMajoramv2
HBASE-20638nightly source artifact testing should fail the stage if it's going to report an error on jiraMajortest
HBASE-20624Race in ReplicationSource which causes walEntryFilter being null when creating new shipperMajorReplication
HBASE-20601Add multiPut support and other miscellaneous to PEMinortooling
HBASE-20627Relocate RS Group pre/post hooks from RSGroupAdminServer to RSGroupAdminEndpointMajor.
HBASE-20591nightly job doesn't respect maven optionsCriticaltest
HBASE-20560Revisit the TestReplicationDroppedTables utMajor.
HBASE-20571JMXJsonServlet generates invalid JSON if it has NaN in metricsMajorUI
HBASE-20585Need to clear peer map when clearing MasterProcedureSchedulerMajorproc-v2
HBASE-20457Return immediately for a scan rpc call when we want to switch from pread to streamMajorscan
HBASE-20447Only fail cacheBlock if block collisions aren't related to next block metadataMajorBlockCache, BucketCache
HBASE-20544downstream HBaseTestingUtility fails with invalid portBlockertest
HBASE-20004Client is not able to execute REST queries in a secure clusterMinorREST, security
HBASE-20475Fix the flaky TestReplicationDroppedTables unit test.Major.
HBASE-20554“WALs outstanding” message from CleanerChore is noisyTrivial.
HBASE-20204Add locking to RefreshFileConnections in BucketCacheMajorBucketCache
HBASE-20485Copy constructor of Scan doesn't copy the readType and replicaIdMinor.
HBASE-20543Fix the flaky TestThriftHttpServerMajor.
HBASE-20521TableOutputFormat.checkOutputSpecs conf checking sequence cause pig script run failMajormapreduce
HBASE-20500[rsgroup] should keep at least one server in default groupMajorrsgroup
HBASE-20517Fix PerformanceEvaluation ‘column’ parameterMajortest
HBASE-20524Need to clear metrics when ReplicationSourceManager refresh replication sourcesMinor.
HBASE-20476Open sequence number could go backwards in AssignProcedureMajorRegion Assignment
HBASE-20506Add doc and test for unused RetryCounter, useful-looking utilityMinor.
HBASE-20492UnassignProcedure is stuck in retry loop on region stuck in OPENING stateCriticalamv2
HBASE-20497The getRecoveredQueueStartPos always return 0 in RecoveredReplicationSourceShipperMajorReplication
HBASE-18842The hbase shell clone_snaphost command returns bad error messageMinorshell
HBASE-20466Consistently use override mechanism for exempt classes in CoprocessClassloaderMajorCoprocessors
HBASE-20006TestRestoreSnapshotFromClientWithRegionReplicas is flakeyCriticalread replicas
HBASE-18059The scanner order for memstore scanners are wrongCriticalregionserver, scan, Scanners
HBASE-20404Ugly cleanerchore complaint that dir is not emptyMajormaster
HBASE-20419Fix potential NPE in ZKUtil#listChildrenAndWatchForNewChildren callersMajor.
HBASE-20364nightly job gives old results or no results for stages that timeout on SCMCriticaltest
HBASE-20335nightly jobs no longer contain machine informationCriticaltest
HBASE-20338WALProcedureStore#recoverLease() should have fixed sleeps for retrying rollWriter()Major.
HBASE-20356make skipping protoc possibleCriticaldependencies, thirdparty
HBASE-15291FileSystem not closed in secure bulkLoadMajor.
HBASE-20068Hadoopcheck project health check uses default maven repo instead of yetus managed onesMajorcommunity, test
HBASE-20361Non-successive TableInputSplits may wrongly be merged by auto balancing featureMajormapreduce
HBASE-20260Purge old content from the book for branch-2/masterCriticaldocumentation
HBASE-20058improper quoting in presplitting command docsMinordocumentation
HBASE-19923Reset peer state and config when refresh replication source failedMajorReplication
HBASE-19748TestRegionReplicaFailover and TestRegionReplicaReplicationEndpoint UT hangsMajor.

TESTS:

JIRASummaryPriorityComponent
HBASE-20667Rename TestGlobalThrottler to TestReplicationGlobalThrottlerTrivial.
HBASE-20646TestWALProcedureStoreOnHDFS failing on branch-1Trivial.
HBASE-20505PE should support multi column family read and write casesMinor.
HBASE-20513Collect and emit ScanMetrics in PerformanceEvaluationMinortest
HBASE-20414TestLockProcedure#testMultipleLocks may fail on slow machineMajor.

SUB-TASKS:

JIRASummaryPriorityComponent
HBASE-20831Copy master doc into branch-2.1 and edit to make it suit 2.1.0Blockerdocumentation
HBASE-20839Fallback to FSHLog if we can not instantiated AsyncFSWAL when user does not specify AsyncFSWAL explicitlyBlockerwal
HBASE-20244NoSuchMethodException when retrieving private method decryptEncryptedDataEncryptionKey from DFSClientBlockerwal
HBASE-20193Basic Replication Web UI - RegionserverCriticalReplication, Usability
HBASE-20489Update Reference Guide that CLUSTER_KEY value is present on the Master UI info page.Minordocumentation
HBASE-19722Meta query statistics metrics sourceMajor.
HBASE-20781Save recalculating families in a WALEdit batch of CellsMajorPerformance
HBASE-20194Basic Replication WebUI - MasterCriticalReplication, Usability
HBASE-20780ServerRpcConnection logging cleanupMajorlogging, Performance
HBASE-19764Fix Checkstyle errors in hbase-endpointMinor.
HBASE-20710extra cloneFamily() in Mutation.add(Cell)Minorregionserver
HBASE-18569Add prefetch support for async region locatorMajorasyncclient, Client
HBASE-20706[hack] Don't add known not-OPEN regions in reopen phase of MTPCriticalamv2
HBASE-20334add a test that expressly uses both our shaded client and the one from hadoop 3Majorhadoop3, shading
HBASE-20615emphasize use of shaded client jars when they're present in an installMajorbuild, Client, Usability
HBASE-20333break up shaded client into one with no Hadoop and one that's standaloneCriticalshading
HBASE-20332shaded mapreduce module shouldn't include hadoopCriticalmapreduce, shading
HBASE-20722Make RegionServerTracker only depend on children changed eventMajor.
HBASE-20700Move meta region when server crash can cause the procedure to be stuckCriticalmaster, proc-v2, Region Assignment
HBASE-20628SegmentScanner does over-comparing when one flushingCriticalPerformance
HBASE-19761Fix Checkstyle errors in hbase-zookeeperMinor.
HBASE-19724Fix Checkstyle errors in hbase-hadoop2-compatMinor.
HBASE-20518Need to serialize the enabled field for UpdatePeerConfigProcedureMajorReplication
HBASE-20481Replicate entries from same region serially in ReplicationEndpoint for serial replicationMajor.
HBASE-20378Provide a hbck option to cleanup replication barrier for a tableMajor.
HBASE-20128Add new UTs which extends the old replication UTs but set replication scope to SERIALMajor.
HBASE-20417Do not read wal entries when peer is disabledMajorReplication
HBASE-20294Also cleanup last pushed sequence id in ReplicationBarrierCleanerMajorReplication
HBASE-20377Deal with table in enabling and disabling state when modifying serial replication peerMajorReplication
HBASE-20367Write a replication barrier for regions when disabling a tableMajorReplication
HBASE-20296Remove last pushed sequence ids when removing tables from a peerMajorReplication
HBASE-20285Delete all last pushed sequence ids when removing a peer or removing the serial flag for a peerMajorReplication
HBASE-20138Find a way to deal with the conflicts when updating replication positionMajorReplication
HBASE-20127Add UT for serial replication after failoverMajorReplication, test
HBASE-20271ReplicationSourceWALReader.switched should use the file name instead of the path object directlyMajorReplication
HBASE-20227Add UT for ReplicationUtils.contains methodMajorReplication, test
HBASE-20147Serial replication will be stuck if we create a table with serial replication but add it to a peer after there are region movesMajor.
HBASE-20116Optimize the region last pushed sequence id layout on zkMajorReplication
HBASE-20242The open sequence number will grow if we fail to open a region after writing the max sequence id fileMajor.
HBASE-20155update branch-2 version to 2.1.0-SNAPSHOTMajorbuild, community
HBASE-20206WALEntryStream should not switch WAL file silentlyMajorReplication
HBASE-20117Cleanup the unused replication barriers in meta tableMajormaster, Replication
HBASE-20165Shell command to make a normal peer to be a serial replication peerMajor.
HBASE-20167Optimize the implementation of ReplicationSourceWALReaderMajorReplication
HBASE-20125Add UT for serial replication after region split and mergeMajorReplication
HBASE-20129Add UT for serial replication checkerMajorReplication
HBASE-20115Reimplement serial replication based on the new replication storage layerMajorReplication
HBASE-20050Reimplement updateReplicationPositions logic in serial replication based on the newly introduced replication storage layerMajor.
HBASE-20082Fix findbugs errors only on master which are introduced by HBASE-19397Majorfindbugs
HBASE-19936Introduce a new base class for replication peer procedureMajor.
HBASE-19719Fix checkstyle issuesMajorproc-v2, Replication
HBASE-19711TestReplicationAdmin.testConcurrentPeerOperations hangsMajorproc-v2
HBASE-19707Race in start and terminate of a replication source after we async start replicatione endpointMajorproc-v2, Replication
HBASE-19636All rs should already start work with the new peer change when replication peer procedure is finishedMajorproc-v2, Replication
HBASE-19634Add permission check for executeProcedures in AccessControllerMajorproc-v2, Replication
HBASE-19697Remove TestReplicationAdminUsingProcedureMajorproc-v2, Replication
HBASE-19661Replace ReplicationStateZKBase with ZKReplicationStorageBaseMajorproc-v2, Replication
HBASE-19687Move the logic in ReplicationZKNodeCleaner to ReplicationChecker and remove ReplicationZKNodeCleanerChoreMajorproc-v2, Replication
HBASE-19544Add UTs for testing concurrent modifications on replication peerMajorproc-v2, Replication, test
HBASE-19686Use KeyLocker instead of ReentrantLock in PeerProcedureHandlerImplMajorproc-v2, Replication
HBASE-19623Create replication endpoint asynchronously when adding a replication sourceMajorproc-v2, Replication
HBASE-19633Clean up the replication queues in the postPeerModification stage when removing a peerMajorproc-v2, Replication
HBASE-19622Reimplement ReplicationPeers with the new replication storage interfaceMajorproc-v2, Replication
HBASE-19635Introduce a thread at RS side to call reportProcedureDoneMajorproc-v2
HBASE-19617Remove ReplicationQueues, use ReplicationQueueStorage directlyMajorReplication
HBASE-19642Fix locking for peer modification procedureCriticalproc-v2, Replication
HBASE-19592Add UTs to test retry on update zk failureMajorproc-v2, Replication
HBASE-19630Add peer cluster key check when add new replication peerMajorproc-v2, Replication
HBASE-19573Rewrite ReplicationPeer with the new replication storage interfaceMajorproc-v2, Replication
HBASE-19579Add peer lock test for shell command list_locksMajorproc-v2, Replication
HBASE-19599Remove ReplicationQueuesClient, use ReplicationQueueStorage directlyMajorReplication
HBASE-19543Abstract a replication storage interface to extract the zk specific codeMajorproc-v2, Replication
HBASE-19525RS side changes for moving peer modification from zk watcher to procedureMajorproc-v2, Replication
HBASE-19580Use slf4j instead of commons-logging in new, just-added Peer Procedure classesMajorproc-v2, Replication
HBASE-19520Add UTs for the new lock type PEERMajorproc-v2
HBASE-19564Procedure id is missing in the response of peer related operationsMajorproc-v2, Replication
HBASE-19536Client side changes for moving peer modification from zk watcher to procedureMajorReplication
HBASE-19524Master side changes for moving peer modification from zk watcher to procedureMajorproc-v2, Replication
HBASE-19216Implement a general framework to execute remote procedure on RSMajorproc-v2, Replication

OTHER:

JIRASummaryPriorityComponent
HBASE-19997[rolling upgrade] 1.x => 2.xBlocker.
HBASE-20360Further optimization for serial replicationMajorReplication
HBASE-20862Address 2.1.0 Compatibility Report IssuesBlockercompatibility
HBASE-20665“Already cached block XXX” message should be DEBUGMinorBlockCache
HBASE-20677Backport test of HBASE-20566 ‘Creating a system table after enabling rsgroup feature puts region into RIT’ to branch-2Major.
HBASE-19475Extend backporting strategy in documentationTrivialdocumentation
HBASE-20595Remove the concept of ‘special tables’ from rsgroupsMajorRegion Assignment, rsgroup
HBASE-20415branches-2 don't need maven-scala-pluginMajorbuild
HBASE-20112Include test results from nightly hadoop3 tests in jenkins test resultsCriticaltest
HBASE-17918document serial replicationCriticaldocumentation, Replication
HBASE-19737Manage a HBASE-19397-branch-2 branch and merge it to branch-2Majorproc-v2, Replication