These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements.
Replace the ForkJoinPool in CleanerChore by ThreadPoolExecutor which can limit the spawn thread size and avoid the master GC frequently. The replacement is an internal implementation in CleanerChore, so no config key change, the upstream users can just upgrade the hbase master without any other change.
Introduced a new config key for the snapshot taking/restoring operations at master side: hbase.master.executor.snapshot.threads, its default value is 3. means we can have 3 snapshot operations running at the same time.
Add serveral API in TimeRange class for avoiding using the deprecated TimeRange constructor: * TimeRange#from: Represents the time interval [minStamp, Long.MAX_VALUE) * TimeRange#until: Represents the time interval [0, maxStamp) * TimeRange#between: Represents the time interval [minStamp, maxStamp)
Provide a public method in MultiRowRangeFilter class to speed the requirement of filtering with multiple row prefixes, it will expand the row prefixes as multiple rowkey ranges by MultiRowRangeFilter, it's more efficient. {code} public MultiRowRangeFilter(byte[][] rowKeyPrefixes); {code}
Update the base docker image to ubuntu 18.04 for the find flaky tests jenkins job.
Adds a fixMeta method to hbck Service. Fixes holes in hbase:meta. Follow-up to fix overlaps.
Follow-on is adding a client-side to hbase-operator-tools that can exploit this new addition (HBASE-22825)
Changes merge so you can merge more than two regions at a time. Currently only available inside HBase. HBASE-22827, a follow-on, is about exposing the facility in the Admin API (and then via the shell).
New shaded artifact for testing: hbase-shaded-testing-util.
We found a critical bug which can lead to WAL corruption when Durability.ASYNC_WAL is used. The reason is that we release a ByteBuffer before actually persist the content into WAL file.
The problem maybe lead to several errors, for example, ArrayIndexOfOutBounds when replaying WAL. This is because that the ByteBuffer is reused by others.
ERROR org.apache.hadoop.hbase.executor.EventHandler: Caught throwable while processing event RS_LOG_REPLAY java.lang.ArrayIndexOutOfBoundsException: 18056 at org.apache.hadoop.hbase.KeyValue.getFamilyLength(KeyValue.java:1365) at org.apache.hadoop.hbase.KeyValue.getFamilyLength(KeyValue.java:1358) at org.apache.hadoop.hbase.PrivateCellUtil.matchingFamily(PrivateCellUtil.java:735) at org.apache.hadoop.hbase.CellUtil.matchingFamily(CellUtil.java:816) at org.apache.hadoop.hbase.wal.WALEdit.isMetaEditFamily(WALEdit.java:143) at org.apache.hadoop.hbase.wal.WALEdit.isMetaEdit(WALEdit.java:148) at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:297) at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:195) at org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:100)
And may even cause segmentation fault and crash the JVM directly. You will see a hs_err_pidXXX.log file and usually the problem is SIGSEGV. This is usually because that the ByteBuffer has already been returned to the OS and used for other purpose.
The problem has been reported several times in the past and this time Wellington Ramos Chevreuil provided the full logs and deeply analyzed the logs so we can find the root cause. And Lijin Bin figured out that the problem may only happen when Durability.ASYNC_WAL is used. Thanks to them.
The problem only effects the 2.x releases, all users are highly recommand to upgrade to a release which has this fix in, especially that if you use Durability.ASYNC_WAL.
Add a new method runHbckChore in Hbck interface and a new shell cmd hbck_chore_run to request HBCK chore to run at master side.
Adds a “CatalogJanitor hbase:meta Consistency Issues” section to the new ‘HBCK Report’ page added by HBASE-22709. This section is empty unless the most recent CatalogJanitor scan turned up problems. If so, will show table of issues found.
When CatalogJanitor runs, it now checks for holes, overlaps, empty info:regioninfo columns and bad servers. Dumps findings into log. Follow-up adds report to new ‘HBCK Report’ linked off the Master UI.
NOTE: All features but the badserver check made it into branch-2.1 and branch-2.0 backports.
This feature is enabled by default. And the hbck chore run per 60 minutes by default. You can config “hbase.master.hbck.checker.interval” to a value lesser than or equal to 0 for disabling the chore.
Notice: the config “hbase.master.hbck.checker.interval” was renamed to “hbase.master.hbck.chore.interval” in HBASE-22737.
Upgrade jackson databind dependency to 2.9.9.1 due to CVEs
https://nvd.nist.gov/vuln/detail/CVE-2019-12814
https://nvd.nist.gov/vuln/detail/CVE-2019-12384
Add a new master web UI to show the potentially problematic opened regions. There are three case:
The config point “hbase.offheapcache.minblocksize” was wrong and is now deprecated. The new config point is “hbase.blockcache.minblocksize”.
OfflineMetaRepair is no longer supported in HBase-2+. Please refer to https://hbase.apache.org/book.html#HBCK2
This tool is deprecated in 2.x and will be removed in 3.0.
In HBASE-20734 we moved the recovered.edits onto the wal file system but when constructing the directory we missed the BASE_NAMESPACE_DIR(‘data’). So when using the default config, you will find that there are lots of new directories at the same level with the ‘data’ directory.
In this issue, we add the BASE_NAMESPACE_DIR back, and also try our best to clean up the wrong directories. But we can only clean up the region level directories, so if you want a clean fs layout on HDFS you still need to manually delete the empty directories at the same level with ‘data’.
The effect versions are 2.2.0, 2.1.[1-5], 1.4.[8-10], 1.3.[3-5].
hbase.regionserver.compaction.check.period is used for controlling how often the compaction checker runs. If unset, will use hbase.server.thread.wakefrequency as default value.
hbase.regionserver.flush.check.period is used for controlling how ofter the flush checker runs. If unset, will use hbase.server.thread.wakefrequency as default value.
Added completebulkload short name for BulkLoadHFilesTool to bin/hbase.
These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements.
The CellUtil.setTimestamp
method changes to be an API with audience LimitedPrivate(COPROC)
in HBase 3.0. With that designation the API should remain stable within a given minor release line, but may change between minor releases.
Previously, this method was deprecated in HBase 2.0 for removal in HBase 3.0. Deprecation messages in HBase 2.y releases have been updated to indicate the expected API audience change.
The class LossyCounting was unintentionally marked Public but was never intended to be part of our public API. This oversight has been corrected and LossyCounting is now marked as Private and going forward may be subject to additional breaking changes or removal without notice. If you have taken a dependency on this class we recommend cloning it locally into your project before upgrading to this release.
Warnings for level headings are corrected in the book for the HBase Incompatibilities section.
Add hadoop 3.0.3, 3.1.1 3.1.2 in our hadoop check jobs.
The DumpReplicationQueues tool will now list replication queues sorted in chronological order.
Fixes a formatting issue in the administration section of the book, where listing indentation were a little bit off.
Removed extra + in HRegion, HStore and LoadIncrementalHFiles for branch-2 and HRegion and HStore for branch-1.
Updated metrics core from 3.2.1 to 3.2.6.
The rubocop definition for the maximum method length was set to 75.
Fixes the formatting of the “Voting on Release Candidates” to actually show the quote and code formatting of the RAT check.
The rubocop configuration in the hbase-shell module now allows a line length with 100 characters, instead of 80 as before. For everything before 2.1.5 this change introduces rubocop itself.
This change allows the system and superusers to initiate compactions, even when a space quota violation policy disallows compactions from happening. The original intent behind disallowing of compactions was to prevent end-user compactions from creating undue I/O load, not disallowing *any* compaction in the system.
Adds new configuration hbase.client.failure.map.cleanup.interval which defaults to ten minutes.
Add a new jenkins file for running pre commit check for GitHub PR.
When insufficient permissions, you now get:
HTTP/1.1 403 Forbidden
on the HTTP side, and in the message
Forbidden org.apache.hadoop.hbase.security.AccessDeniedException: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions for user ‘myuser',action: get, tableName:mytable, family:cf. at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.authorizeAccess(RangerAuthorizationCoprocessor.java:547) and the rest of the ADE stack
Now we will sort the javac WARNING/ERROR before generating diff in pre-commit so we can get a stable output for the error prone. The downside is that we just sort the output lexicographically so the line number will also be sorted lexicographically, which is a bit strange to human.
These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements.
After HBASE-21871, we can specify a peer table name with --peerTableName in VerifyReplication tool like the following: hbase org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication --peerTableName=peerTable 5 TestTable
In addition, we can compare any 2 tables in any remote clusters with specifying both peerId and --peerTableName.
For example: hbase org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication --peerTableName=peerTable zk1,zk2,zk3:2181/hbase TestTable
Change spotbugs version to 3.1.11.
Allows shell to set Scan options previously not exposed. See additions as part of the scan help by typing following hbase shell:
hbase> help ‘scan’
These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements.
We can specify peerQuorumAddress instead of peerId in VerifyReplication tool. So it no longer requires peerId to be setup when using this tool.
For example: hbase org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication zk1,zk2,zk3:2181/hbase testTable
Deprecated HBaseConfiguration#getInt(Configuration, String, String, int) method and removed it from 3.0.0 version.
Introduced an new config key in this issue: hbase.regionserver.inmemory.compaction.pool.size. the default value would be 10. you can configure this to set the pool size of in-memory compaction pool. Note that all memstores in one region server will share the same pool, so if you have many regions in one region server, you need to set this larger to compact faster for better read performance.
Make StoppedRpcClientException extend DoNotRetryIOException.
IMPORTANT: Due to security issues, all users who use hbase thrift should avoid using releases which do not have this fix.
The effect releases are: 2.1.x: 2.1.2 and below 2.0.x: 2.0.4 and below 1.x: 1.4.x and below
If you are using the effect releases above, please consider upgrading to a newer release ASAP.
After HBASE-21620, the filterListWithOR has been a bit slow because we need to merge each sub-filter‘s RC , while before HBASE-21620, we will skip many RC merging, but the logic was wrong. So here we choose another way to optimaze the performance: removing the KeyValueUtil#toNewKeyCell. Anoop Sam John suggested that the KeyValueUtil#toNewKeyCell can save some GC before because if we copy key part of cell into a single byte[], then the block the cell refering won’t be refered by the filter list any more, the upper layer can GC the data block quickly. while after HBASE-21620, we will update the prevCellList for every encountered cell now, so the lifecycle of cell in prevCellList for FilterList will be quite shorter. so just use the cell ref for saving cpu. BTW, we removed all the arrays streams usage in filter list, because it's also quite time-consuming in our test.
We found the memstore snapshotting would cost much time because of calling the time-consuming ConcurrentSkipListMap#Size, it would make the p999 latency spike happen. So in this issue, we remove all ConcurrentSkipListMap#size in memstore by counting the cellsCount in MemstoreSizeing. As the issue described, the p999 latency spike was mitigated.
Now all the Enum configs in ColumnFamilyDescriptor can accept lower case config value.
Python3 support was added to dev-support/submit-patch.py. To install newly required dependencies run `pip install -r dev-support/python-requirements.txt` command.
These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements.
Use de.skuzzle.enforcer.restrict-imports-enforcer-rule extension for maven enforcer plugin to ban illegal imports at compile time. Now if you use illegal imports, for example, import com.google.common.*, there will be a compile error, instead of a checkstyle warning.
Add a sanity check when constructing KeyValue from a byte[]. we use the constructor when we‘re reading kv from socket or HFIle or WAL(replication). the santiy check isn’t designed for discovering the bits corruption in network transferring or disk IO. It is designed to detect bugs inside HBase in advance. and HBASE-21459 indicated that there's extremely small performance loss for diff kinds of keyvalue.
The replication UI on master will show the replication endpoint classname.
Add a SERIAL flag for add_peer command to identifiy whether or not the replication peer is a serial replication peer. The default serial flag is false.
Log level of ReadOnlyZKClient moved to debug.
Allow passing of -Dkey=value option to shell to override hbase-* configuration: e.g.:
$ ./bin/hbase shell -Dhbase.zookeeper.quorum=ZK0.remote.cluster.example.org,ZK1.remote.cluster.example.org,ZK2.remote.cluster.example.org -Draining=false ... hbase(main):001:0> @shell.hbase.configuration.get(“hbase.zookeeper.quorum”) => “ZK0.remote.cluster.example.org,ZK1.remote.cluster.example.org,ZK2.remote.cluster.example.org” hbase(main):002:0> @shell.hbase.configuration.get(“raining”) => “false”
HBase clusters will experience Region Server failures due to out of memory errors due to a leak given any of the following:
When there are long running scans the Region Server process attempts to optimize access by using a different API geared towards sequential access. Due to an error in HBASE-20704 for HBase 2.0+ the Region Server fails to release related resources when those scans finish. That same optimization path is always used for the HBase internal file compaction process.
Impact for this error can be minimized by setting the config value “hbase.storescanner.pread.max.bytes” to MAX_INT to avoid the optimization for default user scans. Clients should also be checked to ensure they do not pass the STREAM read type to the Scan API. This will have a severe impact on performance for long scans.
Compactions always use this sequential optimized reading mechanism so downstream users will need to periodically restart Region Server roles after compactions have happened.
To prevent race condition between in progress snapshot (performed by TakeSnapshotHandler) and HFileCleaner which results in data loss, this JIRA introduced mutual exclusion between taking snapshot and running HFileCleaner. That is, at any given moment, either some snapshot can be taken or, HFileCleaner checks hfiles which are not referenced, but not both can be running.
The procedure for meta table will be executed in a separate worker thread named ‘Urgent Worker’ to avoid stuck. A new config named ‘hbase.master.urgent.procedure.threads’ is added, the default value for it is 1. To disable the separate worker, set it to 0.
Add -Djdk.net.URLClassPath.disableClassPathURLCheck=true when executing surefire plugin.
Use CompatRemoteProcedureResolver instead of ExecuteProceduresRemoteCall to dispatch region open/close requests to RS. Since ExecuteProceduresRemoteCall will group all the open/close operations in one call and execute them sequentially on the target RS. If one operation fails, all the operation will be marked as failure.
Adds scheduleServerCrashProcedure to the HbckService.
These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements.
Adds to bin/hbase means of invoking hbck2. Pass the new ‘-j’ option on the ‘hbck’ command with a value of the full path to the HBCK2.jar.
E.g:
$ ./bin/hbase hbck -j ~/checkouts/hbase-operator-tools/hbase-hbck2/target/hbase-hbck2-1.0.0-SNAPSHOT.jar setTableState x ENABLED
Retry assigns ‘forever’ (or until an intervention such as a ServerCrashProcedure).
Previous retry was a maximum of ten times but on failure, handling was an indeterminate.
The description claims the balancer not dynamically configurable but this is an error; it is http://hbase.apache.org/book.html#dyn_config
Also, if balancer is seen to be cutting out too soon, try setting “hbase.master.balancer.stochastic.runMaxSteps” to true.
Adds cleaner logging around balancer start.
HBASE-21073 | Major | “Maintenance mode” master
Instead of being an ephemeral state set by hbck, maintenance mode is now an explicit toggle set by either configuration property or environment variable. In maintenance mode, master will host system tables and not assign any user-space tables to RSs. This gives operators the ability to affect repairs to meta table with fewer moving parts.
hbase-2.2.x uses a new Procedure form assiging/unassigning/moving Regions; it does not process hbase-2.1.x and earlier Unassign/Assign Procedure types. Upgrade requires that we first drain the Master Procedure Store of old style Procedures before starting the new 2.2.x Master. The patch here facilitates the draining process.
On your running hbase-2.1.1+ (or 2.0.3+ cluster), when upgrading:
Changed waitTime parameter to lockWait on bypass. Changed default waitTime from 0 -- i.e. wait for ever -- to 1ms so if lock is held, we'll go past it and if override enforce bypass.
bypass will now throw an Exception if passed a lockWait <= 0; i.e bypass will prevent an operator getting stuck on an entity lock waiting forever (lockWait == 0)
Cleans up usage and docs around Canary. Does not change command-line args (though we should -- smile).
Scans that make use of QualifierFilter
previously would erroneously return both columns with an empty qualifier along with those that matched. After this change that behavior has changed to only return those columns that match.
This adds two extra features to WALPrettyPrinter tool:
Output for each cell combined size of cell descriptors, plus the cell value itself, in a given WAL edit. This is printed on the results as “cell total size sum:” info by default;
An optional -g/--goto argument, that allows to seek straight to that specific WAL file position, then sequentially reading the WAL from that point towards its end;
Local HBase cluster (as used by unit tests) wait times on startup and initialization can be configured via `hbase.master.start.timeout.localHBaseCluster` and `hbase.master.init.timeout.localHBaseCluster`
Adds anchors #tables, #tasks, etc.
Adds override to assigns and unassigns. Changes bypass ‘force’ to align calling the param ‘override’ instead.
The new override allows ‘overriding’ previous Procedure owner to override their ownership of the Procedured entity (“Region”).
Used by hbck2.
Add table state column to the tables panel
Removed the abort_procedure command from shell -- dangerous -- and deprecated abortProcedure in Admin API.
Added a bypass to hbck2:
{code} $ HBASE_CLASSPATH_PREFIX=../hbase-operator-tools/hbase-hbck2/target/hbase-hbck2-1.0.0-SNAPSHOT.jar ./bin/hbase org.apache.hbase.HBCK2 usage: HBCK2 [OPTIONS] COMMAND <ARGS>
Options: -d,--debug run with debug output -h,--help output this help message -p,--hbase.zookeeper.property.clientPort peerport of target hbase ensemble -q,--hbase.zookeeper.quorum <arg> ensemble of target hbase -z,--zookeeper.znode.parent parent znode of target hbase
Commands: setTableState <TABLENAME> <STATE> Possible table states: ENABLED, DISABLED, DISABLING, ENABLING To read current table state, in the hbase shell run: hbase> get ‘hbase:meta’, ‘<TABLENAME>’, ‘table:state’ A value of \x08\x00 == ENABLED, \x08\x01 == DISABLED, etc. An example making table name ‘user’ ENABLED: $ HBCK2 setTableState users ENABLED Returns whatever the previous table state was.
assigns <ENCODED_REGIONNAME>... A ‘raw’ assign that can be used even during Master initialization. Skirts Coprocessors. Pass one or more encoded RegionNames: e.g. 1588230740 is hard-coded encoding for hbase:meta region and de00010733901a05f5a2a3a382e27dd4 is an example of what a random user-space encoded Region name looks like. For example: $ HBCK2 assign 1588230740 de00010733901a05f5a2a3a382e27dd4 Returns the pid of the created AssignProcedure or -1 if none.
bypass [OPTIONS] <PID>... Pass one (or more) procedure ‘pid’s to skip to the procedure finish. Parent of this procedures will also skip to its finish. Entities will be left in an inconsistent state and will require manual fixup. Pass --force to break any outstanding locks. Pass --waitTime=<seconds> to wait on entity lock before giving up. Default: force=false and waitTime=0. Returns true if succeeded.
unassigns <ENCODED_REGIONNAME>... A ‘raw’ unassign that can be used even during Master initialization. Skirts Coprocessors. Pass one or more encoded RegionNames: Skirts Coprocessors. Pass one or more encoded RegionNames: de00010733901a05f5a2a3a382e27dd4 is an example of what a random user-space encoded Region name looks like. For example: $ HBCK2 unassign 1588230740 de00010733901a05f5a2a3a382e27dd4 Returns the pid of the created UnassignProcedure or -1 if none. {code}
Adds ‘raw’ assigns/unassigns to the Hbck Service. Takes a list of encoded region names and bulk assigns/unassigns. Skirts Master ‘state’ check and does not invoke Coprocessors. For repair only.
Here is what HBCK2 usage looks like now:
{code} $ java -cp hbase-hbck2-1.0.0-SNAPSHOT.jar org.apache.hbase.HBCK2 usage: HBCK2 <OPTIONS> COMMAND [<ARGS>]
Options: -d,--debug run with debug output -h,--help output this help message --hbase.zookeeper.peerport peerport of target hbase ensemble --hbase.zookeeper.quorum ensemble of target hbase --zookeeper.znode.parent parent znode of target hbase
Commands: setTableState <TABLENAME> <STATE> Possible table states: ENABLED, DISABLED, DISABLING, ENABLING To read current table state, in the hbase shell run: hbase> get ‘hbase:meta’, ‘<TABLENAME>’, ‘table:state’ A value of \x08\x00 == ENABLED, \x08\x01 == DISABLED, etc. An example making table name ‘user’ ENABLED: $ HBCK2 setTableState users ENABLED Returns whatever the previous table state was.
assign <ENCODED_REGIONNAME> ... A ‘raw’ assign that can be used even during Master initialization. Skirts Coprocessors. Pass one or more encoded RegionNames: e.g. 1588230740 is hard-coded encoding for hbase:meta region and de00010733901a05f5a2a3a382e27dd4 is an example of what a random user-space encoded Region name looks like. For example: $ HBCK2 assign 1588230740 de00010733901a05f5a2a3a382e27dd4 Returns the pid of the created AssignProcedure or -1 if none.
unassign <ENCODED_REGIONNAME> ... A ‘raw’ unassign that can be used even during Master initialization. Skirts Coprocessors. Pass one or more encoded RegionNames: Skirts Coprocessors. Pass one or more encoded RegionNames: de00010733901a05f5a2a3a382e27dd4 is an example of what a random user-space encoded Region name looks like. For example: $ HBCK2 unassign 1588230740 de00010733901a05f5a2a3a382e27dd4 Returns the pid of the created UnassignProcedure or -1 if none. {code}
Puts master startup into holding pattern if meta is not assigned (previous it would exit). To make progress again, operator needs to inject an assign (Caveats and instruction can be found in HBASE-21035).
Allows configuration of the length of RPC messages printed to the log at TRACE level via “hbase.ipc.trace.param.size” in RpcServer.
Make it so can run the WAL parse and load system in isolation. Here is an example:
{code}$ HBASE_OPTS=" -XX:+UnlockDiagnosticVMOptions -XX:+UnlockCommercialFeatures -XX:+FlightRecorder -XX:+DebugNonSafepoints" ./bin/hbase org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore ~/big_set_of_masterprocwals/ {code}
Client facing artifacts are now built whenever Maven is run through the “package” goal. Previously, the client facing artifacts would create placeholder jars that skipped repackaging HBase and third-party dependencies unless the “release” profile was active.
Build times may be noticeably longer depending on your build hardware. For example, the Jenkins worker nodes maintained by ASF Infra take ~14% longer to do a full packaging build. An example portability-focused personal laptop took ~25% longer.
Users who have previously made use of prefix tree encoding can now check that their existing HFiles no longer contain data that uses it with an additional preupgrade check command.
hbase pre-upgrade validate-hfile
Please see the “HFile Content validation” section of the ref guide's coverage of the pre-upgrade validator tool for usage details.
Adds an HBCK Service and a first method to force-change-in-table-state for use by an HBCK client effecting ‘repair’ to a malfunctioning HBase.
Fence out hbase-1.x hbck1 instances. Stop them making state changes on an hbase-2.x cluster; they could do damage. We do this by writing the hbck1 lock file into place on hbase-2.x Master start-up.
To disable this new behavior, set hbase.write.hbck1.lock.file to false
HFiles generated by 2.0.0, 2.0.1, 2.1.0 are not forward compatible to 1.4.6-, 1.3.2.1-, 1.2.6.1-, and other inactive releases. Why HFile lose compatability is hbase in new versions (2.0.0, 2.0.1, 2.1.0) use protobuf to serialize/deserialize TimeRangeTracker (TRT) while old versions use DataInput/DataOutput. To solve this, We have to put HBASE-21012 to 2.x and put HBASE-21013 in 1.x. For more information, please check HBASE-21008.
After HBASE-20965, we can use MasterFifoRpcScheduler in master to separate RegionServerReport requests to indenpedent handler. To use this feature, please set “hbase.master.rpc.scheduler.factory.class” to “org.apache.hadoop.hbase.ipc.MasterFifoRpcScheduler”. Use “hbase.master.server.report.handler.count” to set RegionServerReport handlers count, the default value is half of “hbase.regionserver.handler.count” value, but at least 1, and the other handlers count in master is “hbase.regionserver.handler.count” value minus RegionServerReport handlers count, but at least 1 too.
In previous releases, when a Space Quota was configured on a table or namespace and that table or namespace was deleted, the Space Quota was also deleted. This change improves the implementation so that the same is also done for RPC Quotas.
After HBASE-20986, we can set different value to block size of WAL and recovered edits. Both of their default value is 2 * default HDFS blocksize. And hbase.regionserver.recoverededits.blocksize is for block size of recovered edits while hbase.regionserver.hlog.blocksize is for block size of WAL.
With this change if a WAL's meta provider (hbase.wal.meta_provider) is not explicitly set, it now defaults to whatever hbase.wal.provider is set to. Previous, the two settings operated independently, each with its own default.
This change is operationally incompatible with previous HBase versions because the default WAL meta provider no longer defaults to AsyncFSWALProvider but to hbase.wal.provider.
The thought is that this is more in line with an operator's expectation, that a change in hbase.wal.provider is sufficient to change how WALs are written, especially given hbase.wal.meta_provider is an obscure configuration and that the very idea that meta regions would have their own wal provider would likely come as a surprise.
Update hadoop-two.version to 2.7.7 and hadoop-three.version to 3.0.3 due to a JDK issue which is solved by HADOOP-15473.
Make hasLock method final, and add a locked field in Procedure to record whether we have the lock. We will set it to true in doAcquireLock and to false in doReleaseLock. The sub procedures do not need to manage it any more.
Also added a locked field in the proto message. When storing, the field will be set according to the return value of hasLock. And when loading, there is a new field in Procedure called lockedWhenLoading. We will set it to true if the locked field in proto message is true.
The reason why we can not set the locked field directly to true by calling doAcquireLock is that, during initialization, most procedures need to wait until master is initialized. So the solution here is that, we introduced a new method called waitInitialized in Procedure, and move the wait master initialized related code from acquireLock to this method. And we added a restoreLock method to Procedure, if lockedWhenLoading is true, we will call the acquireLock to get the lock, but do not set locked to true. And later when we call doAcquireLock and pass the waitInitialized check, we will test lockedWhenLoading, if it is true, when we just set the locked field to true and return, without actually calling the acquireLock method since we have already called it once.
Class org.apache.hadoop.hbase.util.Base64 has been removed in it's entirety from HBase 2+. In HBase 1, unused methods have been removed from the class and the audience was changed from Public to Private. This class was originally intended as an internal utility class that could be used externally but thinking since changed; these classes should not have been advertised as public to end-users.
This represents an incompatible change for users who relied on this implementation. An alternative implementation for affected clients is available at java.util.Base64 when using Java 8 or newer; be aware, it may encode/decode differently. For clients seeking to restore this specific implementation, it is available in the public domain for download at http://iharder.sourceforge.net/current/java/base64/
These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements.
After HBASE-20691 we have changed the default setting of hbase.wal.storage.policy from “HOT” back to “NONE” which means we defer the policy to HDFS. This fixes the problem of release 2.0.0 that the storage policy of WAL directory will defer to HDFS and may not be “HOT” even if you explicitly set hbase.wal.storage.policy to “HOT”
As we hack into the internal of DFSClient when implementing AsyncFSWAL to get better performance, a patch release of hadoop can make it broken.
So now, if user does not specify a wal provider, then we will first try to use ‘asyncfs’, i.e, the AsyncFSWALProvider. If we fail due to some compatible issues, we will fallback to ‘filesystem’, i.e, FSHLog.
After HBASE-20193, we add a section to web ui to show the replication status of each wal group. There are 2 parts of this section, they both show the peerId, wal group and current replicating log of each replication source. And one is showing the information of replication log queue, i.e. size of current log, log queue size and replicating offset. The other one is showing the delay of replication, i.e. last shipped age and replication delay. If the offset shows -1 and replication delay is UNKNOWN, that means replication is not started. This may be caused by this peer is disabled or the replicationEndpoint is sleeping due to some reason.
Now we have a ‘basically work’ solution for rolling upgrade from 1.4.x to 2.x. Please see the “Rolling Upgrade from 1.x to 2.x” section in ref guide for more details.
The command help that followed all errors, before, is now no longer available. Erroneous command inputs would now just show error-texts followed by the shell command to try for seeing the help message. It looks like: For usage try 'help “create”’. Operators can copy-paste the command to get the help message.
After HBASE-20194, we added 2 parts to master‘s web page. One is Peers that shows all replication peers and some of their configurations, like peer id, cluster key, state, bandwidth, and which namespace or table it will replicate. The other one is replication status of all regionservers, we added a tab to region servers division, then we can check the replication delay of all region servers for any peer. This table shows AgeOfLastShippedOp, SizeOfLogQueue and ReplicationLag for each regionserver and the table is sort by ReplicationLag in descending order. By this way we can easily find the problematic region server. If the replication delay is UNKNOWN, that means this walGroup doesn’t start replicate yet and it may get disabled. ReplicationLag will update once this peer start replicate.
Add prefetch support for async region locator. The default value is 10. Set ‘hbase.client.locate.prefetch.limit’ in hbase-site.xml if you want to use another value for it.
This changes client-side nonce generation to use the same nonce for re-submissions of client RPC DDL operations.
Introduce an InitMetaProcedure to initialize meta table for a new HBase deploy. Marked RecoverMetaProcedure deprecated and remove the usage of it in the current code base. We still need to keep it in place for compatibility. The code in RecoverMetaProcedure has been moved to ServerCrashProcedure, and SCP will always be enabled and we will rely on it to bring meta region online.
For more on the issue addressed by this commit, see the design doc for overview and plan: https://docs.google.com/document/d/1_872oHzrhJq4ck7f6zmp1J--zMhsIFvXSZyX1Mxg5MA/edit#heading=h.xy1z4alsq7uy
HBase now includes a helper script that can be used to run a basic functionality test for a given HBase installation at in dev_support
. The test can optionally be given an HBase client artifact to rely on and can optionally be given specific Hadoop client artifacts to use.
For usage information see ./dev-support/hbase_nightly_pseudo-distributed-test.sh --help
.
The project nightly tests now make use of this test to check running on top of Hadoop 2, Hadoop 3, and Hadoop 3 with shaded client artifacts.
The HBase convenience binary artifacts now includes a client focused tarball that a) includes more docs and b) does not include scripts or jars only needed for running HBase cluster services.
The new artifact is made as a normal part of the assembly:single
maven command.
HBase's built in scripts now rely on the downstream facing shaded artifacts where possible. In particular interest to downstream users, the hbase classpath
and hbase mapredcp
commands now return the relevant shaded client artifact and only those third paty jars needed to make use of them (e.g. slf4j-api, commons-logging, htrace, etc).
Downstream users should note that by default the hbase classpath
command will treat having hadoop
on the shell's PATH as an implicit request to include the output of the hadoop classpath
command in the returned classpath. This long-existing behavior can be opted out of by setting the environment variable HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP
to the value “true”. For example: HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP="true" bin/hbase classpath
.
Downstream users who need to use both HBase and Hadoop APIs should switch to relying on the new hbase-shaded-client-byo-hadoop
artifact rather than the existing hbase-shaded-client
artifact. The new artifact no longer includes and Hadoop classes.
It should work in combination with either the output of hadoop classpath
or the Hadoop provided client-facing shaded artifacts in Hadoop 3+.
The hbase-shaded-mapreduce
artifact no longer include its own copy of Hadoop classes. Users who make use of the artifact via YARN should be able to get these classes from YARN's classpath without having to make any changes.
Users of our integration tests on Hadoop 3 can now add all needed dependencies by pointing at jars included in our binary convenience artifact.
Prior to this fix, downstream users on Hadoop 3 would need to get a copy of the Hamcrest v1.3 jar from elsewhere.
Adds two new properties for hbase-site.xml for THRIFT SPNEGO when in HTTP mode: * hbase.thrift.spnego.keytab.file * hbase.thrift.spnego.principal
Adds a negotiation logic between a secure java REST client and server. After this jira the Java REST client will start responding to the Negotiate challenge sent by the server. Adds RESTDemoClient which can be used to verify whether the secure Java REST client works against secure REST server or not.
A second attempt at fixing HBASE-20173. Fixes unfinished keeping of server state inside AM (ONLINE=>SPLITTING=>OFFLINE=>null). Concurrent unassigns look at server state to figure if they should wait on SCP to wake them up or not.
This patch adds an FSUtil.copyFilesParallel() to help copy files in parallel, and it will return all the paths of directories and files traversed. Thus when we copy manifest in ExportSnapshot, we can copy reference files concurrently and use the paths it returns to help setOwner and setPermission. The size of thread pool is determined by the configuration snapshot.export.copy.references.threads, and its default value is the number of runtime available processors.
Before this change we would incorrectly include the size of enqueued store files for bulk replication in the calculation for determining whether or not to rate limit the transfer of WAL edits. Because bulk replication uses a separate and asynchronous mechanism for file transfer this could incorrectly limit the batch sizes for WAL replication if bulk replication in progress, with negative impact on latency and throughput.
PreUpgradeValidator tool with DataBlockEncoding validator was added to verify cluster is upgradable to HBase 2.
HBase is no longer able to maintain compatibility with Apache Hadoop versions that are no longer receiving updates. This release raises the minimum supported version to Hadoop 2.7.1. Downstream users are strongly advised to upgrade to the latest Hadoop 2.7 maintenance release.
Downstream users of earlier HBase versions are similarly advised to upgrade to Hadoop 2.7.1+. When doing so, it is especially important to follow the guidance from the HBase Reference Guide's Hadoop section on replacing the Hadoop artifacts bundled with HBase.
Add multiPut support Set --multiPut=number to enable batchput(meanwhile, --autoflush need be set to false)
Add Connection Count support Added a new parameter connCount to PE. set --connCount=2 means all threads will share 2 connections. oneCon option and connCount option shouldn't be set at the same time.
Add avg RT and avg TPS/QPS statstic for all threads
Delete some redundant code Now RandomWriteTest is inherited from SequentialWrite.
HBase now relies on an internal mechanism to determine when it is running a local hbase cluster meant for external interaction vs an encapsulated test. When created via the HBaseTestingUtility
, ports for Master and RegionServer services and UIs will be set to random ports to allow for multiple parallel uses on a single machine. Normally when running a Standalone HBase Deployment (as described in the HBase Reference Guide) the ports will be picked according to the same defaults used in a full cluster set up. If you wish to instead use the random port assignment set hbase.localcluster.assign.random.ports
to true.
Added ‘hbase.rest.http.allow.options.method’ configuration property to allow user to decide whether Rest Server HTTP should allow OPTIONS method or not. By default it is enabled in HBase 2.1.0+ versions and in other versions it is disabled. Similarly ‘hbase.thrift.http.allow.options.method’ is added HBase 1.5, 2.1.0 and 3.0.0 versions. It is disabled by default.
This change will enable users to perform append and increment operation with null qualifier via hbase-shell.
When attempting to clone a snapshot but using a namespace that does not exist, the HBase shell will now correctly report the exception as caused by the passed namespace. Previously, the shell would report that the problem was an unknown namespace but it would claim the user provided table name was not found as a namespace. Both before and after this change the shell properly used the passed namespace to attempt to handle the request.
When configured to do thrift-over-http, the HBase Thrift API Server no longer accepts the HTTP methods TRACE nor OPTIONS.
Now in replication we can make sure the order of pushing logs is same as the order of requests from client. Set the serial flag to true for a replication peer to enable this feature.
After HBASE-20159 we allow client to use different ZK quorums by introducing three new properties: hbase.client.zookeeper.quorum and hbase.client.zookeeper.property.clientPort to specify client zookeeper properties (note that the combination of these two properties should be different from the server ZK quorums), and hbase.client.zookeeper.observer.mode to indicate whether the client ZK nodes are in observer mode (false by default)
HConstants.DEFAULT_ZOOKEPER_CLIENT_PORT has been removed in HBase 3.0 and replaced by the correctly spelled DEFAULT_ZOOKEEPER_CLIENT_PORT.
Now when opening a region, we will store the current max sequence id of the region to its max sequence id file instead of the ‘next sequence id’. This could avoid the sequence id bumping when we fail to open a region, and also align to the behavior when we close a region.
The default durability setting for the synchronous WAL is Durability.SYNC_WAL, which triggers HDFS hflush() to flush edits to the datanodes. We also support Durability.FSYNC_WAL, which instead triggers HDFS hsync() to flush _and_ fsync edits. This change introduces the new configuration setting “hbase.wal.hsync”, defaulting to FALSE, that if set to TRUE changes the default durability setting for the synchronous WAL to FSYNC_WAL.
After HBASE-19389 we introduced a RegionServer self-protection mechanism to prevent write handler getting exhausted by high concurrency put with dense columns, mainly through two new properties: hbase.region.store.parallel.put.limit.min.column.count to decide what kind of put (with how many columns within a single column family) to limit (100 by default) and hbase.region.store.parallel.put.limit to limit the concurrency (10 by default). There's another property for advanced user and please check source and javadoc of StoreHotnessProtector for more details.
A new method setSerial has been added to the interface ReplicationPeerConfigBuilder which is marked as IA.Public. This interface is not supposed to be implemented by client code, but if you do, this will be an incompatible change as you need to add this method to your implementation too.
Introduce 5 procedures to do peer modifications: AddPeerProcedure RemovePeerProcedure UpdatePeerConfigProcedure EnablePeerProcedure DisablePeerProcedure
The procedures are all executed with the following stage:
The procedure will hold an exclusive lock on the peer id, so now there is no concurrent modifications on a single peer.
And now it is guaranteed that once the procedure is done, the peer modification has already taken effect on all RSes.
Abstracte a storage layer for replication peer/queue manangement, and refactored the upper layer to remove zk related naming/code/comment.
Add pre/postExecuteProcedures CP hooks to RegionServerObserver, and add permission check for executeProcedures method which requires the caller to be system user or super user.
On rolling upgrade: just do not do any replication peer modifications during the rolling upgrading. There is no pb/layout changes on the peer/queue storage on zk.