////
/**
 *
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
////

[[upgrading]]
= Upgrading
:doctype: book
:numbered:
:toc: left
:icons: font
:experimental:

You cannot skip major versions when upgrading. If you are upgrading from version 0.98.x to 2.x, you must first go from 0.98.x to 1.2.x and then go from 1.2.x to 2.x.

Review <<configuration>>, in particular <<hadoop>>. Familiarize yourself with <<hbase_supported_tested_definitions>>.

[[hbase.versioning]]
== HBase version number and compatibility

[[hbase.versioning.post10]]
=== Aspirational Semantic Versioning

Starting with the 1.0.0 release, HBase is working towards link:http://semver.org/[Semantic Versioning] for its release versioning. In summary:

.Given a version number MAJOR.MINOR.PATCH, increment the:
* MAJOR version when you make incompatible API changes,
* MINOR version when you add functionality in a backwards-compatible manner, and
* PATCH version when you make backwards-compatible bug fixes.
* Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

[[hbase.versioning.compat]]
.Compatibility Dimensions
In addition to the usual API versioning considerations HBase has other compatibility dimensions that we need to consider.

.Client-Server wire protocol compatibility
* Allows updating client and server out of sync.
* We could only allow upgrading the server first. I.e. the server would be backward compatible to an old client, that way new APIs are OK.
* Example: A user should be able to use an old client to connect to an upgraded cluster.

.Server-Server protocol compatibility
* Servers of different versions can co-exist in the same cluster.
* The wire protocol between servers is compatible.
* Workers for distributed tasks, such as replication and log splitting, can co-exist in the same cluster.
* Dependent protocols (such as using ZK for coordination) will also not be changed.
* Example: A user can perform a rolling upgrade.

.File format compatibility
* Support file formats backward and forward compatible
* Example: File, ZK encoding, directory layout is upgraded automatically as part of an HBase upgrade. User can downgrade to the older version and everything will continue to work.

.Client API compatibility
* Allow changing or removing existing client APIs.
* An API needs to be deprecated for a whole major version before we will change/remove it.
** An example: An API was deprecated in 2.0.1 and will be marked for deletion in 4.0.0. On the other hand, an API deprecated in 2.0.0 can be removed in 3.0.0.
** Occasionally mistakes are made and internal classes are marked with a higher access level than they should. In these rare circumstances, we will accelerate the deprecation schedule to the next major version (i.e., deprecated in 2.2.x, marked `IA.Private` 3.0.0). Such changes are communicated and explained via release note in Jira.
* APIs available in a patch version will be available in all later patch versions. However, new APIs may be added which will not be available in earlier patch versions.
* New APIs introduced in a patch version will only be added in a source compatible way footnote:[See 'Source Compatibility' https://blogs.oracle.com/darcy/entry/kinds_of_compatibility]: i.e. code that implements public APIs will continue to compile.
** Example: A user using a newly deprecated API does not need to modify application code with HBase API calls until the next major version.
*

.Client Binary compatibility
* Client code written to APIs available in a given patch release can run unchanged (no recompilation needed) against the new jars of later patch versions.
* Client code written to APIs available in a given patch release might not run against the old jars from an earlier patch version.
** Example: Old compiled client code will work unchanged with the new jars.
* If a Client implements an HBase Interface, a recompile MAY be required upgrading to a newer minor version (See release notes
for warning about incompatible changes). All effort will be made to provide a default implementation so this case should not arise.

.Server-Side Limited API compatibility (taken from Hadoop)
* Internal APIs are marked as Stable, Evolving, or Unstable
* This implies binary compatibility for coprocessors and plugins (pluggable classes, including replication) as long as these are only using marked interfaces/classes.
* Example: Old compiled Coprocessor, Filter, or Plugin code will work unchanged with the new jars.

.Dependency Compatibility
* An upgrade of HBase will not require an incompatible upgrade of a dependent project, except for Apache Hadoop.
* An upgrade of HBase will not require an incompatible upgrade of the Java runtime.
* Example: Upgrading HBase to a version that supports _Dependency Compatibility_ won't require that you upgrade your Apache ZooKeeper service.
* Example: If your current version of HBase supported running on JDK 8, then an upgrade to a version that supports _Dependency Compatibility_ will also run on JDK 8.

.Hadoop Versions
[TIP]
====
Previously, we tried to maintain dependency compatibility for the underly Hadoop service but over the last few years this has proven untenable. While the HBase project attempts to maintain support for older versions of Hadoop, we drop the "supported" designator for minor versions that fail to continue to see releases. Additionally, the Hadoop project has its own set of compatibility guidelines, which means in some cases having to update to a newer supported minor release might break some of our compatibility promises.
====

.Operational Compatibility
* Metric changes
* Behavioral changes of services
* JMX APIs exposed via the `/jmx/` endpoint

.Summary
* A patch upgrade is a drop-in replacement. Any change that is not Java binary and source compatible would not be allowed.footnote:[See http://docs.oracle.com/javase/specs/jls/se7/html/jls-13.html.] Downgrading versions within patch releases may not be compatible.

* A minor upgrade requires no application/client code modification. Ideally it would be a drop-in replacement but client code, coprocessors, filters, etc might have to be recompiled if new jars are used.

* A major upgrade allows the HBase community to make breaking changes.

.Compatibility Matrix footnote:[Note that this indicates what could break, not that it will break. We will/should add specifics in our release notes.]
[cols="1,1,1,1"]
|===
| | Major | Minor | Patch
|Client-Server wire Compatibility|  N |Y |Y
|Server-Server Compatibility |N |Y |Y
|File Format Compatibility | N footnote:[comp_matrix_offline_upgrade_note,Running an offline upgrade tool without downgrade might be needed. We will typically only support migrating data from major version X to major version X+1.] | Y |Y
|Client API Compatibility  | N | Y |Y
|Client Binary Compatibility | N | N |Y
4+|Server-Side Limited API Compatibility
>| Stable | N | Y | Y
>| Evolving | N |N |Y
>| Unstable | N |N |N
|Dependency Compatibility | N |Y |Y
|Operational Compatibility | N |N |Y
|===

[IMPORTANT]
======
HBase 1.7.0 release violated client-server wire compatibility guarantees and was subsequently
withdrawn after the incompatibilities were reported and fixed in 1.7.1. If you are considering an
upgrade to 1.7.x line, see <<upgrade1.7.x>>.
======

[[hbase.client.api.surface]]
==== HBase API Surface

HBase has a lot of API points, but for the compatibility matrix above, we differentiate between Client API, Limited Private API, and Private API. HBase uses link:https://yetus.apache.org/documentation/in-progress/interface-classification/[Apache Yetus Audience Annotations] to guide downstream expectations for stability.

* InterfaceAudience (link:https://yetus.apache.org/documentation/in-progress/javadocs/org/apache/yetus/audience/InterfaceAudience.html[javadocs]): captures the intended audience, possible values include:
  - Public: safe for end users and external projects
  - LimitedPrivate: used for internals we expect to be pluggable, such as coprocessors
  - Private: strictly for use within HBase itself
Classes which are defined as `IA.Private` may be used as parameters or return values for interfaces which are declared `IA.LimitedPrivate`. Treat the `IA.Private` object as opaque; do not try to access its methods or fields directly.
* InterfaceStability (link:https://yetus.apache.org/documentation/in-progress/javadocs/org/apache/yetus/audience/InterfaceStability.html[javadocs]): describes what types of interface changes are permitted. Possible values include:
  - Stable: the interface is fixed and is not expected to change
  - Evolving: the interface may change in future minor verisons
  - Unstable: the interface may change at any time

Please keep in mind the following interactions between the `InterfaceAudience` and `InterfaceStability` annotations within the HBase project:

* `IA.Public` classes are inherently stable and adhere to our stability guarantees relating to the type of upgrade (major, minor, or patch).
* `IA.LimitedPrivate` classes should always be annotated with one of the given `InterfaceStability` values. If they are not, you should presume they are `IS.Unstable`.
* `IA.Private` classes should be considered implicitly unstable, with no guarantee of stability between releases.

[[hbase.client.api]]
HBase Client API::
  HBase Client API consists of all the classes or methods that are marked with InterfaceAudience.Public interface. All main classes in hbase-client and dependent modules have either InterfaceAudience.Public, InterfaceAudience.LimitedPrivate, or InterfaceAudience.Private marker. Not all classes in other modules (hbase-server, etc) have the marker. If a class is not annotated with one of these, it is assumed to be a InterfaceAudience.Private class.

[[hbase.limitetprivate.api]]
HBase LimitedPrivate API::
  LimitedPrivate annotation comes with a set of target consumers for the interfaces. Those consumers are coprocessors, phoenix, replication endpoint implementations or similar. At this point, HBase only guarantees source and binary compatibility for these interfaces between patch versions.

[[hbase.private.api]]
HBase Private API::
  All classes annotated with InterfaceAudience.Private or all classes that do not have the annotation are for HBase internal use only. The interfaces and method signatures can change at any point in time. If you are relying on a particular interface that is marked Private, you should open a jira to propose changing the interface to be Public or LimitedPrivate, or an interface exposed for this purpose.

[[hbase.binary.compatibility]]
.Binary Compatibility
When we say two HBase versions are compatible, we mean that the versions are wire and binary compatible. Compatible HBase versions means that clients can talk to compatible but differently versioned servers. It means too that you can just swap out the jars of one version and replace them with the jars of another, compatible version and all will just work. Unless otherwise specified, HBase point versions are (mostly) binary compatible. You can safely do rolling upgrades between binary compatible versions; i.e. across maintenance releases: e.g. from 1.4.4 to 1.4.6. See link:[Does compatibility between versions also mean binary compatibility?] discussion on the HBase dev mailing list.

[[hbase.rolling.upgrade]]
=== Rolling Upgrades

A rolling upgrade is the process by which you update the servers in your cluster a server at a time. You can rolling upgrade across HBase versions if they are binary or wire compatible. See <<hbase.rolling.restart>> for more on what this means. Coarsely, a rolling upgrade is a graceful stop each server, update the software, and then restart. You do this for each server in the cluster. Usually you upgrade the Master first and then the RegionServers. See <<rolling>> for tools that can help use the rolling upgrade process.

For example, in the below, HBase was symlinked to the actual HBase install. On upgrade, before running a rolling restart over the cluster, we changed the symlink to point at the new HBase software version and then ran

[source,bash]
----
$ HADOOP_HOME=~/hadoop-2.6.0-CRC-SNAPSHOT ~/hbase/bin/rolling-restart.sh --config ~/conf_hbase
----

The rolling-restart script will first gracefully stop and restart the master, and then each of the RegionServers in turn. Because the symlink was changed, on restart the server will come up using the new HBase version. Check logs for errors as the rolling upgrade proceeds.

[[hbase.rolling.restart]]
.Rolling Upgrade Between Versions that are Binary/Wire Compatible
Unless otherwise specified, HBase minor versions are binary compatible. You can do a <<hbase.rolling.upgrade>> between HBase point versions. For example, you can go to 1.4.4 from 1.4.6 by doing a rolling upgrade across the cluster replacing the 1.4.4 binary with a 1.4.6 binary.

In the minor version-particular sections below, we call out where the versions are wire/protocol compatible and in this case, it is also possible to do a <<hbase.rolling.upgrade>>.

== Rollback

Sometimes things don't go as planned when attempting an upgrade. This section explains how to perform a _rollback_ to an earlier HBase release. Note that this should only be needed between Major and some Minor releases. You should always be able to _downgrade_ between HBase Patch releases within the same Minor version. These instructions may require you to take steps before you start the upgrade process, so be sure to read through this section beforehand.

=== Caveats

.Rollback vs Downgrade
This section describes how to perform a _rollback_ on an upgrade between HBase minor and major versions. In this document, rollback refers to the process of taking an upgraded cluster and restoring it to the old version _while losing all changes that have occurred since upgrade_. By contrast, a cluster _downgrade_ would restore an upgraded cluster to the old version while maintaining any data written since the upgrade. We currently only offer instructions to rollback HBase clusters. Further, rollback only works when these instructions are followed prior to performing the upgrade.

When these instructions talk about rollback vs downgrade of prerequisite cluster services (i.e. HDFS), you should treat leaving the service version the same as a degenerate case of downgrade.

.Replication
Unless you are doing an all-service rollback, the HBase cluster will lose any configured peers for HBase replication. If your cluster is configured for HBase replication, then prior to following these instructions you should document all replication peers. After performing the rollback you should then add each documented peer back to the cluster. For more information on enabling HBase replication, listing peers, and adding a peer see <<hbase.replication.management>>. Note also that data written to the cluster since the upgrade may or may not have already been replicated to any peers. Determining which, if any, peers have seen replication data as well as rolling back the data in those peers is out of the scope of this guide.

.Data Locality
Unless you are doing an all-service rollback, going through a rollback procedure will likely destroy all locality for Region Servers. You should expect degraded performance until after the cluster has had time to go through compactions to restore data locality. Optionally, you can force a compaction to speed this process up at the cost of generating cluster load.

.Configurable Locations
The instructions below assume default locations for the HBase data directory and the HBase znode. Both of these locations are configurable and you should verify the value used in your cluster before proceeding. In the event that you have a different value, just replace the default with the one found in your configuration
* HBase data directory is configured via the key 'hbase.rootdir' and has a default value of '/hbase'.
* HBase znode is configured via the key 'zookeeper.znode.parent' and has a default value of '/hbase'.

=== All service rollback

If you will be performing a rollback of both the HDFS and ZooKeeper services, then HBase's data will be rolled back in the process.

.Requirements

* Ability to rollback HDFS and ZooKeeper

.Before upgrade
No additional steps are needed pre-upgrade. As an extra precautionary measure, you may wish to use distcp to back up the HBase data off of the cluster to be upgraded. To do so, follow the steps in the 'Before upgrade' section of 'Rollback after HDFS downgrade' but copy to another HDFS instance instead of within the same instance.

.Performing a rollback

. Stop HBase
. Perform a rollback for HDFS and ZooKeeper (HBase should remain stopped)
. Change the installed version of HBase to the previous version
. Start HBase
. Verify HBase contents—use the HBase shell to list tables and scan some known values.

=== Rollback after HDFS rollback and ZooKeeper downgrade

If you will be rolling back HDFS but going through a ZooKeeper downgrade, then HBase will be in an inconsistent state. You must ensure the cluster is not started until you complete this process.

.Requirements

* Ability to rollback HDFS
* Ability to downgrade ZooKeeper

.Before upgrade
No additional steps are needed pre-upgrade. As an extra precautionary measure, you may wish to use distcp to back up the HBase data off of the cluster to be upgraded. To do so, follow the steps in the 'Before upgrade' section of 'Rollback after HDFS downgrade' but copy to another HDFS instance instead of within the same instance.

.Performing a rollback

. Stop HBase
. Perform a rollback for HDFS and a downgrade for ZooKeeper (HBase should remain stopped)
. Change the installed version of HBase to the previous version
. Clean out ZooKeeper information related to HBase. WARNING: This step will permanently destroy all replication peers. Please see the section on HBase Replication under Caveats for more information.
+
.Clean HBase information out of ZooKeeper
[source,bash]
----
[hpnewton@gateway_node.example.com ~]$ zookeeper-client -server zookeeper1.example.com:2181,zookeeper2.example.com:2181,zookeeper3.example.com:2181
Welcome to ZooKeeper!
JLine support is disabled
rmr /hbase
quit
Quitting...
----
. Start HBase
. Verify HBase contents—use the HBase shell to list tables and scan some known values.

=== Rollback after HDFS downgrade

If you will be performing an HDFS downgrade, then you'll need to follow these instructions regardless of whether ZooKeeper goes through rollback, downgrade, or reinstallation.

.Requirements

* Ability to downgrade HDFS
* Pre-upgrade cluster must be able to run MapReduce jobs
* HDFS super user access
* Sufficient space in HDFS for at least two copies of the HBase data directory

.Before upgrade
Before beginning the upgrade process, you must take a complete backup of HBase's backing data. The following instructions cover backing up the data within the current HDFS instance. Alternatively, you can use the distcp command to copy the data to another HDFS cluster.

. Stop the HBase cluster
. Copy the HBase data directory to a backup location using the https://hadoop.apache.org/docs/current/hadoop-distcp/DistCp.html[distcp command] as the HDFS super user (shown below on a security enabled cluster)
+
.Using distcp to backup the HBase data directory
[source,bash]
----

[hpnewton@gateway_node.example.com ~]$ kinit -k -t hdfs.keytab hdfs@EXAMPLE.COM
[hpnewton@gateway_node.example.com ~]$ hadoop distcp /hbase /hbase-pre-upgrade-backup

----
. Distcp will launch a mapreduce job to handle copying the files in a distributed fashion. Check the output of the distcp command to ensure this job completed successfully.

.Performing a rollback

. Stop HBase
. Perform a downgrade for HDFS and a downgrade/rollback for ZooKeeper (HBase should remain stopped)
. Change the installed version of HBase to the previous version
. Restore the HBase data directory from prior to the upgrade as the HDFS super user (shown below on a security enabled cluster). If you backed up your data on another HDFS cluster instead of locally, you will need to use the distcp command to copy it back to the current HDFS cluster.
+
.Restore the HBase data directory
[source,bash]
----
[hpnewton@gateway_node.example.com ~]$ kinit -k -t hdfs.keytab hdfs@EXAMPLE.COM
[hpnewton@gateway_node.example.com ~]$ hdfs dfs -mv /hbase /hbase-upgrade-rollback
[hpnewton@gateway_node.example.com ~]$ hdfs dfs -mv /hbase-pre-upgrade-backup /hbase
----
. Clean out ZooKeeper information related to HBase. WARNING: This step will permanently destroy all replication peers. Please see the section on HBase Replication under Caveats for more information.
+
.Clean HBase information out of ZooKeeper
[source,bash]
----
[hpnewton@gateway_node.example.com ~]$ zookeeper-client -server zookeeper1.example.com:2181,zookeeper2.example.com:2181,zookeeper3.example.com:2181
Welcome to ZooKeeper!
JLine support is disabled
rmr /hbase
quit
Quitting...
----
. Start HBase
. Verify HBase contents–use the HBase shell to list tables and scan some known values.

== Upgrade Paths
[[upgrade3.0]]
=== Upgrade from 2.x to 3.x
The RegionServer Grouping feature has been reimplemented. See section
<<migrating.rsgroup>> in <<ops_mgt>> for more details.

[[upgrade2.3]]
=== Upgrade from 2.0.x-2.2.x to 2.3+
There is no special consideration upgrading to hbase-2.3.x from earlier versions. From 2.2.x, it should be
rolling upgradeable. From 2.1.x or 2.0.x, you will need to clear the <<upgrade2.2>> hurdle first.

[[upgrade2.3_zookeeper]]
==== Upgraded ZooKeeper Dependency Version

Our dependency on Apache ZooKeeper has been upgraded to 3.5.7
(https://issues.apache.org/jira/browse/HBASE-24132[HBASE-24132]), as 3.4.x is EOL. The newer 3.5.x
client is compatible with the older 3.4.x server. However, if you're using HBase in stand-alone
mode and perform an in-place upgrade, there are some upgrade steps
https://cwiki.apache.org/confluence/display/ZOOKEEPER/Upgrade+FAQ[documented by the ZooKeeper community].
This doesn't impact a production deployment, but would impact a developer's local environment.

[[upgrade2.3_in-master-procedure-store-region]]
==== New In-Master Procedure Store

Of note, HBase 2.3.0 changes the in-Master Procedure Store implementation. It was a dedicated custom store
(see <<master.wal>>) to instead use a standard HBase Region (https://issues.apache.org/jira/browse/HBASE-23326[HBASE-23326]).
The migration from the old to new format is automatic run by the new 2.3.0 Master on startup. The old _MasterProcWALs_
dir which hosted the old custom implementation files in _${hbase.rootdir}_ is deleted on successful
migration. A new _MasterProc_ sub-directory replaces it to host the Store files and WALs for the new
Procedure Store in-Master Region. The in-Master Region is unusual in that it writes to an
alternate location at _${hbase.rootdir}/MasterProc_ rather than under _${hbase.rootdir}/data_ in the
filesystem and the special Procedure Store in-Master Region is hidden from all clients other than the active
Master itself. Otherwise, it is like any other with the Master process running flushes and compactions,
archiving WALs when over-flushed, and so on. Its files are readable by standard Region and Store file
tooling for triage and analysis as long as they are pointed to the appropriate location in the filesystem.

[[upgrade2.2]]
=== Upgrade from 2.0 or 2.1 to 2.2+

HBase 2.2+ uses a new Procedure form assiging/unassigning/moving Regions. It does not process HBase 2.1 and 2.0's Unassign/Assign Procedure types. Upgrade requires that we first drain the Master Procedure Store of old style Procedures before starting the new 2.2 Master. So you need to make sure that before you kill the old version (2.0 or 2.1) Master, there is no region in transition. And once the new version (2.2+) Master is up, you can rolling upgrade RegionServers one by one.

And there is a more safer way if you are running 2.1.1+ or 2.0.3+ cluster. It need four steps to upgrade Master.

. Shutdown both active and standby Masters (Your cluster will continue to server reads and writes without interruption).
. Set the property hbase.procedure.upgrade-to-2-2 to true in hbase-site.xml for the Master, and start only one Master, still using the 2.1.1+ (or 2.0.3+) version.
. Wait until the Master quits. Confirm that there is a 'READY TO ROLLING UPGRADE' message in the Master log as the cause of the shutdown. The Procedure Store is now empty.
. Start new Masters with the new 2.2+ version.

Then you can rolling upgrade RegionServers one by one. See link:https://issues.apache.org/jira/browse/HBASE-21075[HBASE-21075] for more details.

[[upgrade2.0]]
=== Upgrading from 1.x to 2.x

In this section we will first call out significant changes compared to the prior stable HBase release and then go over the upgrade process. Be sure to read the former with care so you avoid suprises.

==== Changes of Note!

First we'll cover deployment / operational changes that you might hit when upgrading to HBase 2.0+. After that we'll call out changes for downstream applications. Please note that Coprocessors are covered in the operational section. Also note that this section is not meant to convey information about new features that may be of interest to you. For a complete summary of changes, please see the CHANGES.txt file in the source release artifact for the version you are planning to upgrade to.

[[upgrade2.0.basic.requirements]]
.Update to basic prerequisite minimums in HBase 2.0+
As noted in the section <<basic.prerequisites>>, HBase 2.0+ requires a minimum of Java 8 and Hadoop 2.6. The HBase community recommends ensuring you have already completed any needed upgrades in prerequisites prior to upgrading your HBase version.

[[upgrade2.0.hbck]]
.HBCK must match HBase server version
You *must not* use an HBase 1.x version of HBCK against an HBase 2.0+ cluster. HBCK is strongly tied to the HBase server version. Using the HBCK tool from an earlier release against an HBase 2.0+ cluster will destructively alter said cluster in unrecoverable ways.

As of HBase 2.0, HBCK (A.K.A _HBCK1_ or _hbck1_) is a read-only tool that can report the status of some non-public system internals but will often misread state because it does not understand the workings of hbase2.

To read about HBCK's replacement, see <<HBCK2>> in <<ops_mgt>>.

IMPORTANT: Related, before you upgrade, ensure that _hbck1_ reports no `INCONSISTENCIES`. Fixing hbase1-type inconsistencies post-upgrade is an involved process.

////
Link to a ref guide section on HBCK in 2.0 that explains use and calls out the inability of clients and server sides to detect version of each other.
////

[[upgrade2.0.removed.configs]]
.Configuration settings no longer in HBase 2.0+

The following configuration settings are no longer applicable or available. For details, please see the detailed release notes.

* hbase.config.read.zookeeper.config (see <<upgrade2.0.zkconfig>> for migration details)
* hbase.zookeeper.useMulti (HBase now always uses ZK's multi functionality)
* hbase.rpc.client.threads.max
* hbase.rpc.client.nativetransport
* hbase.fs.tmp.dir
// These next two seem worth a call out section?
* hbase.bucketcache.combinedcache.enabled
* hbase.bucketcache.ioengine no longer supports the 'heap' value.
* hbase.bulkload.staging.dir
* hbase.balancer.tablesOnMaster wasn't removed, strictly speaking, but its meaning has fundamentally changed and users should not set it. See the section <<upgrade2.0.regions.on.master>> for details.
* hbase.master.distributed.log.replay See the section <<upgrade2.0.distributed.log.replay>> for details
* hbase.regionserver.disallow.writes.when.recovering See the section <<upgrade2.0.distributed.log.replay>> for details
* hbase.regionserver.wal.logreplay.batch.size See the section <<upgrade2.0.distributed.log.replay>> for details
* hbase.master.catalog.timeout
* hbase.regionserver.catalog.timeout
* hbase.metrics.exposeOperationTimes
* hbase.metrics.showTableName
* hbase.online.schema.update.enable (HBase now always supports this)
* hbase.thrift.htablepool.size.max

[[upgrade2.0.renamed.configs]]
.Configuration properties that were renamed in HBase 2.0+

The following properties have been renamed. Attempts to set the old property will be ignored at run time.

.Renamed properties
[options="header"]
|============================================================================================================
|Old name |New name
|hbase.rpc.server.nativetransport |hbase.netty.nativetransport
|hbase.netty.rpc.server.worker.count |hbase.netty.worker.count
|hbase.hfile.compactions.discharger.interval |hbase.hfile.compaction.discharger.interval
|hbase.hregion.percolumnfamilyflush.size.lower.bound |hbase.hregion.percolumnfamilyflush.size.lower.bound.min
|============================================================================================================

[[upgrade2.0.changed.defaults]]
.Configuration settings with different defaults in HBase 2.0+

The following configuration settings changed their default value. Where applicable, the value to set to restore the behavior of HBase 1.2 is given.

* hbase.security.authorization now defaults to false. set to true to restore same behavior as previous default.
* hbase.client.retries.number is now set to 10. Previously it was 35. Downstream users are advised to use client timeouts as described in section <<config_timeouts>> instead.
* hbase.client.serverside.retries.multiplier is now set to 3. Previously it was 10. Downstream users are advised to use client timesout as describe in section <<config_timeouts>> instead.
* hbase.master.fileSplitTimeout is now set to 10 minutes. Previously it was 30 seconds.
* hbase.regionserver.logroll.multiplier is now set to 0.5. Previously it was 0.95. This change is tied with the following doubling of block size. Combined, these two configuration changes should make for WALs of about the same size as those in hbase-1.x but there should be less incidence of small blocks because we fail to roll the WAL before we hit the blocksize threshold. See link:https://issues.apache.org/jira/browse/HBASE-19148[HBASE-19148] for discussion.
* hbase.regionserver.hlog.blocksize defaults to 2x the HDFS default block size for the WAL dir. Previously it was equal to the HDFS default block size for the WAL dir.
* hbase.client.start.log.errors.counter changed to 5. Previously it was 9.
* hbase.ipc.server.callqueue.type changed to 'fifo'. In HBase versions 1.0 - 1.2 it was 'deadline'. In prior and later 1.x versions it already defaults to 'fifo'.
* hbase.hregion.memstore.chunkpool.maxsize is 1.0 by default. Previously it was 0.0. Effectively, this means previously we would not use a chunk pool when our memstore is onheap and now we will. See the section <<gcpause>> for more infromation about the MSLAB chunk pool.
* hbase.master.cleaner.interval is now set to 10 minutes. Previously it was 1 minute.
* hbase.master.procedure.threads will now default to 1/4 of the number of available CPUs, but not less than 16 threads. Previously it would be number of threads equal to number of CPUs.
* hbase.hstore.blockingStoreFiles is now 16. Previously it was 10.
* hbase.http.max.threads is now 16. Previously it was 10.
* hbase.client.max.perserver.tasks is now 2. Previously it was 5.
* hbase.normalizer.period is now 5 minutes. Previously it was 30 minutes.
* hbase.regionserver.region.split.policy is now SteppingSplitPolicy. Previously it was IncreasingToUpperBoundRegionSplitPolicy.
* replication.source.ratio is now 0.5. Previously it was 0.1.

[[upgrade2.0.regions.on.master]]
."Master hosting regions" feature broken and unsupported

The feature "Master acts as region server" and associated follow-on work available in HBase 1.y is non-functional in HBase 2.y and should not be used in a production setting due to deadlock on Master initialization. Downstream users are advised to treat related configuration settings as experimental and the feature as inappropriate for production settings.

A brief summary of related changes:

* Master no longer carries regions by default
* hbase.balancer.tablesOnMaster is a boolean, default false (if it holds an HBase 1.x list of tables, will default to false)
* hbase.balancer.tablesOnMaster.systemTablesOnly is boolean to keep user tables off master. default false
* those wishing to replicate old list-of-servers config should deploy a stand-alone RegionServer process and then rely on Region Server Groups

[[upgrade2.0.distributed.log.replay]]
."Distributed Log Replay" feature broken and removed

The Distributed Log Replay feature was broken and has been removed from HBase 2.y+. As a consequence all related configs, metrics, RPC fields, and logging have also been removed. Note that this feature was found to be unreliable in the run up to HBase 1.0, defaulted to being unused, and was effectively removed in HBase 1.2.0 when we started ignoring the config that turns it on (link:https://issues.apache.org/jira/browse/HBASE-14465[HBASE-14465]). If you are currently using the feature, be sure to perform a clean shutdown, ensure all DLR work is complete, and disable the feature prior to upgrading.

[[upgrade2.0.prefix-tree.removed]]
._prefix-tree_ encoding removed

The prefix-tree encoding was removed from HBase 2.0.0 (link:https://issues.apache.org/jira/browse/HBASE-19179[HBASE-19179]).
It was (late!) deprecated in hbase-1.2.7, hbase-1.4.0, and hbase-1.3.2.

This feature was removed because it as not being actively maintained. If interested in reviving this
sweet facility which improved random read latencies at the expensive of slowed writes,
write the HBase developers list at _dev at hbase dot apache dot org_.

The prefix-tree encoding needs to be removed from all tables before upgrading to HBase 2.0+.
To do that first you need to change the encoding from PREFIX_TREE to something else that is supported in HBase 2.0.
After that you have to major compact the tables that were using PREFIX_TREE encoding before.
To check which column families are using incompatible data block encoding you can use <<ops.pre-upgrade,Pre-Upgrade Validator>>.

[[upgrade2.0.metrics]]
.Changed metrics

The following metrics have changed names:

* Metrics previously published under the name "AssignmentManger" [sic] are now published under the name "AssignmentManager"

The following metrics have changed their meaning:

* The metric 'blockCacheEvictionCount' published on a per-region server basis no longer includes blocks removed from the cache due to the invalidation of the hfiles they are from (e.g. via compaction).
* The metric 'totalRequestCount' increments once per request; previously it incremented by the number of `Actions` carried in the request; e.g. if a request was a `multi` made of four Gets and two Puts, we'd increment 'totalRequestCount' by six; now we increment by one regardless. Expect to see lower values for this metric in hbase-2.0.0.
* The 'readRequestCount' now counts reads that return a non-empty row where in older hbases, we'd increment 'readRequestCount' whether a Result or not. This change will flatten the profile of the read-requests graphs if requests for non-existent rows. A YCSB read-heavy workload can do this dependent on how the database was loaded.

The following metrics have been removed:

* Metrics related to the Distributed Log Replay feature are no longer present. They were previsouly found in the region server context under the name 'replay'. See the section <<upgrade2.0.distributed.log.replay>> for details.

The following metrics have been added:

* 'totalRowActionRequestCount' is a count of region row actions summing reads and writes.

[[upgrade2.0.logging]]
.Changed logging
HBase-2.0.0 now uses link:https://www.slf4j.org/[slf4j] as its logging frontend.
Prevously, we used link:http://logging.apache.org/log4j/1.2/[log4j (1.2)].
For most the transition should be seamless; slf4j does a good job interpreting
_log4j.properties_ logging configuration files such that you should not notice
any difference in your log system emissions.

That said, your _log4j.properties_ may need freshening. See link:https://issues.apache.org/jira/browse/HBASE-20351[HBASE-20351]
for example, where a stale log configuration file manifest as netty configuration
being dumped at DEBUG level as preamble on every shell command invocation.

[[upgrade2.0.zkconfig]]
.ZooKeeper configs no longer read from zoo.cfg

HBase no longer optionally reads the 'zoo.cfg' file for ZooKeeper related configuration settings. If you previously relied on the 'hbase.config.read.zookeeper.config' config for this functionality, you should migrate any needed settings to the hbase-site.xml file while adding the prefix 'hbase.zookeeper.property.' to each property name.

[[upgrade2.0.permissions]]
.Changes in permissions
The following permission related changes either altered semantics or defaults:

* Permissions granted to a user now merge with existing permissions for that user, rather than over-writing them. (see link:https://issues.apache.org/jira/browse/HBASE-17472[the release note on HBASE-17472] for details)
* Region Server Group commands (added in 1.4.0) now require admin privileges.

[[upgrade2.0.admin.commands]]
.Most Admin APIs don't work against an HBase 2.0+ cluster from pre-HBase 2.0 clients

A number of admin commands are known to not work when used from a pre-HBase 2.0 client. This includes an HBase Shell that has the library jars from pre-HBase 2.0. You will need to plan for an outage of use of admin APIs and commands until you can also update to the needed client version.

The following client operations do not work against HBase 2.0+ cluster when executed from a pre-HBase 2.0 client:

* list_procedures
* split
* merge_region
* list_quotas
* enable_table_replication
* disable_table_replication
* Snapshot related commands

.Deprecated in 1.0 admin commands have been removed.

The following commands that were deprecated in 1.0 have been removed. Where applicable the replacement command is listed.

* The 'hlog' command has been removed. Downstream users should rely on the 'wal' command instead.

[[upgrade2.0.memory]]
.Region Server memory consumption changes.

Users upgrading from versions prior to HBase 1.4 should read the instructions in section <<upgrade1.4.memory>>.

Additionally, HBase 2.0 has changed how memstore memory is tracked for flushing decisions. Previously, both the data size and overhead for storage were used to calculate utilization against the flush threashold. Now, only data size is used to make these per-region decisions. Globally the addition of the storage overhead is used to make decisions about forced flushes.

[[upgrade2.0.ui.splitmerge.by.row]]
.Web UI for splitting and merging operate on row prefixes

Previously, the Web UI included functionality on table status pages to merge or split based on an encoded region name. In HBase 2.0, instead this functionality works by taking a row prefix.

[[upgrade2.0.replication]]
.Special upgrading for Replication users from pre-HBase 1.4

User running versions of HBase prior to the 1.4.0 release that make use of replication should be sure to read the instructions in the section <<upgrade1.4.replication>>.

[[upgrade2.0.shell]]
.HBase shell changes

The HBase shell command relies on a bundled JRuby instance. This bundled JRuby been updated from version 1.6.8 to version 9.1.10.0. The represents a change from Ruby 1.8 to Ruby 2.3.3, which introduces non-compatible language changes for user scripts.

The HBase shell command now ignores the '--return-values' flag that was present in early HBase 1.4 releases. Instead the shell always behaves as though that flag were passed. If you wish to avoid having expression results printed in the console you should alter your IRB configuration as noted in the section <<irbrc>>.

[[upgrade2.0.coprocessors]]
.Coprocessor APIs have changed in HBase 2.0+

All Coprocessor APIs have been refactored to improve supportability around binary API compatibility for future versions of HBase. If you or applications you rely on have custom HBase coprocessors, you should read link:https://issues.apache.org/jira/browse/HBASE-18169[the release notes for HBASE-18169] for details of changes you will need to make prior to upgrading to HBase 2.0+.

For example, if you had a BaseRegionObserver in HBase 1.2 then at a minimum you will need to update it to implement both RegionObserver and RegionCoprocessor and add the method

[source,java]
----
...
  @Override
  public Optional<RegionObserver> getRegionObserver() {
    return Optional.of(this);
  }
...
----

////
This would be a good place to link to a coprocessor migration guide
////

[[upgrade2.0.hfile3.only]]
.HBase 2.0+ can no longer write HFile v2 files.

HBase has simplified our internal HFile handling. As a result, we can no longer write HFile versions earlier than the default of version 3. Upgrading users should ensure that hfile.format.version is not set to 2 in hbase-site.xml before upgrading. Failing to do so will cause Region Server failure. HBase can still read HFiles written in the older version 2 format.

[[upgrade2.0.pb.wal.only]]
.HBase 2.0+ can no longer read Sequence File based WAL file.

HBase can no longer read the deprecated WAL files written in the Apache Hadoop Sequence File format. The hbase.regionserver.hlog.reader.impl and hbase.regionserver.hlog.writer.impl configuration entries should be set to use the Protobuf based WAL reader / writer classes. This implementation has been the default since HBase 0.96, so legacy WAL files should not be a concern for most downstream users.

A clean cluster shutdown should ensure there are no WAL files. If you are unsure of a given WAL file's format you can use the `hbase wal` command to parse files while the HBase cluster is offline. In HBase 2.0+, this command will not be able to read a Sequence File based WAL. For more information on the tool see the section <<hlog_tool.prettyprint>>.

[[upgrade2.0.filters]]
.Change in behavior for filters

The Filter ReturnCode NEXT_ROW has been redefined as skipping to next row in current family, not to next row in all family. it’s more reasonable, because ReturnCode is a concept in store level, not in region level.

[[upgrade2.0.shaded.client.preferred]]
.Downstream HBase 2.0+ users should use the shaded client
Downstream users are strongly urged to rely on the Maven coordinates org.apache.hbase:hbase-shaded-client for their runtime use. This artifact contains all the needed implementation details for talking to an HBase cluster while minimizing the number of third party dependencies exposed.

Note that this artifact exposes some classes in the org.apache.hadoop package space (e.g. o.a.h.configuration.Configuration) so that we can maintain source compatibility with our public API. Those classes are included so that they can be altered to use the same relocated third party dependencies as the rest of the HBase client code. In the event that you need to *also* use Hadoop in your code, you should ensure all Hadoop related jars precede the HBase client jar in your classpath.

[[upgrade2.0.mapreduce.module]]
.Downstream HBase 2.0+ users of MapReduce must switch to new artifact
Downstream users of HBase's integration for Apache Hadoop MapReduce must switch to relying on the org.apache.hbase:hbase-shaded-mapreduce module for their runtime use. Historically, downstream users relied on either the org.apache.hbase:hbase-server or org.apache.hbase:hbase-shaded-server artifacts for these classes. Both uses are no longer supported and in the vast majority of cases will fail at runtime.

Note that this artifact exposes some classes in the org.apache.hadoop package space (e.g. o.a.h.configuration.Configuration) so that we can maintain source compatibility with our public API. Those classes are included so that they can be altered to use the same relocated third party dependencies as the rest of the HBase client code. In the event that you need to *also* use Hadoop in your code, you should ensure all Hadoop related jars precede the HBase client jar in your classpath.

[[upgrade2.0.dependencies]]
.Significant changes to runtime classpath
A number of internal dependencies for HBase were updated or removed from the runtime classpath. Downstream client users who do not follow the guidance in <<upgrade2.0.shaded.client.preferred>> will have to examine the set of dependencies Maven pulls in for impact. Downstream users of LimitedPrivate Coprocessor APIs will need to examine the runtime environment for impact. For details on our new handling of third party libraries that have historically been a problem with respect to harmonizing compatible runtime versions, see the reference guide section <<thirdparty>>.

[[upgrade2.0.public.api]]
.Multiple breaking changes to source and binary compatibility for client API
The Java client API for HBase has a number of changes that break both source and binary compatibility for details see the Compatibility Check Report for the release you'll be upgrading to.

[[upgrade2.0.tracing]]
.Tracing implementation changes
The backing implementation of HBase's tracing features was updated from Apache HTrace 3 to HTrace 4, which includes several breaking changes. While HTrace 3 and 4 can coexist in the same runtime, they will not integrate with each other, leading to disjoint trace information.

The internal changes to HBase during this upgrade were sufficient for compilation, but it has not been confirmed that there are no regressions in tracing functionality. Please consider this feature experimental for the immediate future.

If you previously relied on client side tracing integrated with HBase operations, it is recommended that you upgrade your usage to HTrace 4 as well.

After the Apache HTrace project moved to the Attic/retired, the traces in HBase are left broken and unmaintained since HBase 2.0. A new project link:https://issues.apache.org/jira/browse/HBASE-22120[HBASE-22120] will replace HTrace with OpenTelemetry. It will be shipped in 3.0.0 release. Please see the reference guide section <<tracing>> for more details.

[[upgrade2.0.hfile.compatability]]
.HFile lose forward compatability

HFiles generated by 2.0.0, 2.0.1, 2.1.0 are not forward compatible to 1.4.6-, 1.3.2.1-, 1.2.6.1-,
and other inactive releases. Why HFile lose compatability is hbase in new versions
(2.0.0, 2.0.1, 2.1.0) use protobuf to serialize/deserialize TimeRangeTracker (TRT) while old
versions use DataInput/DataOutput. To solve this, We have to put
link:https://jira.apache.org/jira/browse/HBASE-21012[HBASE-21012]
to 2.x and put link:https://jira.apache.org/jira/browse/HBASE-21013[HBASE-21013] in 1.x.
For more information, please check
link:https://jira.apache.org/jira/browse/HBASE-21008[HBASE-21008].

[[upgrade2.0.perf]]
.Performance

You will likely see a change in the performance profile on upgrade to hbase-2.0.0 given
read and write paths have undergone significant change. On release, writes may be
slower with reads about the same or much better, dependent on context. Be prepared
to spend time re-tuning (See <<performance>>).
Performance is also an area that is now under active review so look forward to
improvement in coming releases (See
link:https://issues.apache.org/jira/browse/HBASE-20188[HBASE-20188 TESTING Performance]).

[[upgrade2.0.it.kerberos]]
.Integration Tests and Kerberos
Integration Tests (`IntegrationTests*`) used to rely on the Kerberos credential cache
for authentication against secured clusters. This used to lead to tests failing due
to authentication failures when the tickets in the credential cache expired.
As of hbase-2.0.0 (and hbase-1.3.0+), the integration test clients will make use
of the configuration properties `hbase.client.keytab.file` and
`hbase.client.kerberos.principal`. They are required. The clients will perform a
login from the configured keytab file and automatically refresh the credentials
in the background for the process lifetime (See
link:https://issues.apache.org/jira/browse/HBASE-16231[HBASE-16231]).

[[upgrade2.0.compaction.throughput.limit]]
.Default Compaction Throughput
HBase 2.x comes with default limits to the speed at which compactions can execute. This
limit is defined per RegionServer. In previous versions of HBase earlier than 1.5, there
was no limit to the speed at which a compaction could run by default. Applying a limit
to the throughput of a compaction should ensure more stable operations from RegionServers.

Take care to notice that this limit is _per RegionServer_, not _per compaction_.

The throughput limit is defined as a range of bytes written per second, and is
allowed to vary within the given lower and upper bound. RegionServers observe the
current throughput of a compaction and apply a linear formula to adjust the allowed
throughput, within the lower and upper bound, with respect to external pressure.
For compactions, external pressure is defined as the number of store files with
respect to the maximum number of allowed store files. The more store files, the
higher the compaction pressure.

Configuration of this throughput is governed by the following properties.

- The lower bound is defined by `hbase.hstore.compaction.throughput.lower.bound`
  and defaults to 50 MB/s (`52428800`).
- The upper bound is defined by `hbase.hstore.compaction.throughput.higher.bound`
  and defaults to 100 MB/s (`104857600`).

To revert this behavior to the unlimited compaction throughput of earlier versions
of HBase, please set the following property to the implementation that applies no
limits to compactions.

`hbase.regionserver.throughput.controller=org.apache.hadoop.hbase.regionserver.throttle.NoLimitThroughputController`

////
This would be a good place to link to an appendix on migrating applications
////

[[upgrade2.0.coprocessors]]
==== Upgrading Coprocessors to 2.0
Coprocessors have changed substantially in 2.0 ranging from top level design changes in class
hierarchies to changed/removed methods, interfaces, etc.
(Parent jira: link:https://issues.apache.org/jira/browse/HBASE-18169[HBASE-18169 Coprocessor fix
and cleanup before 2.0.0 release]). Some of the reasons for such widespread changes:

. Pass Interfaces instead of Implementations; e.g. TableDescriptor instead of HTableDescriptor and
Region instead of HRegion (link:https://issues.apache.org/jira/browse/HBASE-18241[HBASE-18241]
Change client.Table and client.Admin to not use HTableDescriptor).
. Design refactor so implementers need to fill out less boilerplate and so we can do more
compile-time checking (link:https://issues.apache.org/jira/browse/HBASE-17732[HBASE-17732])
. Purge Protocol Buffers from Coprocessor API
(link:https://issues.apache.org/jira/browse/HBASE-18859[HBASE-18859],
link:https://issues.apache.org/jira/browse/HBASE-16769[HBASE-16769], etc)
. Cut back on what we expose to Coprocessors removing hooks on internals that were too private to
 expose (for eg. link:https://issues.apache.org/jira/browse/HBASE-18453[HBASE-18453]
 CompactionRequest should not be exposed to user directly;
 link:https://issues.apache.org/jira/browse/HBASE-18298[HBASE-18298] RegionServerServices Interface
 cleanup for CP expose; etc)

To use coprocessors in 2.0, they should be rebuilt against new API otherwise they will fail to
load and HBase processes will die.

Suggested order of changes to upgrade the coprocessors:

. Directly implement observer interfaces instead of extending Base*Observer classes. Change
 `Foo extends BaseXXXObserver` to `Foo implements XXXObserver`.
 (link:https://issues.apache.org/jira/browse/HBASE-17312[HBASE-17312]).
. Adapt to design change from Inheritence to Composition
 (link:https://issues.apache.org/jira/browse/HBASE-17732[HBASE-17732]) by following
 link:https://github.com/apache/hbase/blob/master/dev-support/design-docs/Coprocessor_Design_Improvements-Use_composition_instead_of_inheritance-HBASE-17732.adoc#migrating-existing-cps-to-new-design[this
 example].
. getTable() has been removed from the CoprocessorEnvrionment, coprocessors should self-manage
 Table instances.

Some examples of writing coprocessors with new API can be found in hbase-example module
link:https://github.com/apache/hbase/tree/branch-2.0/hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example[here] .

Lastly, if an api has been changed/removed that breaks you in an irreparable way, and if there's a
good justification to add it back, bring it our notice (dev@hbase.apache.org).

[[upgrade2.0.rolling.upgrades]]
==== Rolling Upgrade from 1.x to 2.x

Rolling upgrades are currently an experimental feature.
They have had limited testing. There are likely corner
cases as yet uncovered in our
limited experience so you should be careful if you go this
route. The stop/upgrade/start described in the next section,
<<upgrade2.0.process>>, is the safest route.

That said, the below is a prescription for a
rolling upgrade of a 1.4 cluster.

.Pre-Requirements
* Upgrade to the latest 1.4.x release. Pre 1.4 releases may also work but are not tested, so please upgrade to 1.4.3+ before upgrading to 2.x, unless you are an expert and familiar with the region assignment and crash processing. See the section <<upgrade1.4>> on how to upgrade to 1.4.x.
* Make sure that the zk-less assignment is enabled, i.e, set `hbase.assignment.usezk` to `false`. This is the most important thing. It allows the 1.x master to assign/unassign regions to/from 2.x region servers. See the release note section of link:https://issues.apache.org/jira/browse/HBASE-11059[HBASE-11059] on how to migrate from zk based assignment to zk less assignment.
* Before you upgrade, ensure that _hbck1_ reports no `INCONSISTENCIES`. Fixing hbase1-type inconsistencies post-upgrade is an involved process.
* We have tested rolling upgrading from 1.4.3 to 2.1.0, but it should also work if you want to upgrade to 2.0.x.

.Instructions
. Unload a region server and upgrade it to 2.1.0. With link:https://issues.apache.org/jira/browse/HBASE-17931[HBASE-17931] in place, the meta region and regions for other system tables will be moved to this region server immediately. If not, please move them manually to the new region server. This is very important because
** The schema of meta region is hard coded, if meta is on an old region server, then the new region servers can not access it as it does not have some families, for example, table state.
** Client with lower version can communicate with server with higher version, but not vice versa. If the meta region is on an old region server, the new region server will use a client with higher version to communicate with a server with lower version, this may introduce strange problems.
. Rolling upgrade all other region servers.
. Upgrading masters.

It is OK that during the rolling upgrading there are region server crashes. The 1.x master can assign regions to both 1.x and 2.x region servers, and link:https://issues.apache.org/jira/browse/HBASE-19166[HBASE-19166] fixed a problem so that 1.x region server can also read the WALs written by 2.x region server and split them.

NOTE: please read the <<Changes of Note!,Changes of Note!>> section carefully before rolling upgrading. Make sure that you do not use the removed features in 2.0, for example, the prefix-tree encoding, the old hfile format, etc. They could both fail the upgrading and leave the cluster in an intermediate state and hard to recover.

NOTE: If you have success running this prescription, please notify the dev list with a note on your experience and/or update the above with any deviations you may have taken so others going this route can benefit from your efforts.

[[upgrade2.0.process]]
==== Upgrade process from 1.x to 2.x

To upgrade an existing HBase 1.x cluster, you should:

* Ensure that _hbck1_ reports no `INCONSISTENCIES`. Fixing hbase1-type inconsistencies post-upgrade is an involved process. Fix all _hbck1_ complaints before proceeding.
* Clean shutdown of existing 1.x cluster
* Update coprocessors
* Upgrade Master roles first
* Upgrade RegionServers
* (Eventually) Upgrade Clients

[[upgrade1.7.x]]
=== Upgrading to 1.7.1+
HBase release 1.7.0 introduced an incompatible table metadata serialization format that broke the
minor release compatibility guarantees. The issue was reported in https://issues.apache.org/jira/browse/HBASE-26021[HBASE-26021] and the problematic serialization patch was reverted in
HBase 1.7.1. Some important notes about 1.7.x upgrades below.

* If you are considering an upgrade to 1.7.x version, skip 1.7.0 completely and upgrade to 1.7.1+
version. 1.7.0 was withdrawn and removed from the Apache sites.
* If you already installed a 1.7.0 cluster from scratch and are looking to migrate to 1.7.1+, you
cannot follow the regular rolling upgrade procedures due to broken compatibility contracts.
Instead shutdown the cluster and reboot with 1.7.1+ binaries. Newer versions detect any existing
tables with incompatible serialization and rewrite them using the correct format at bootstrap.
* If you are already on 1.7.1+ version, everything is good and no additional steps need to be
performed.

[[upgrade1.4]]
=== Upgrading from pre-1.4 to 1.4+

[[upgrade1.4.memory]]
==== Region Server memory consumption changes.

Users upgrading from versions prior to HBase 1.4 should be aware that the estimates of heap usage by the memstore objects (KeyValue, object and array header sizes, etc) have been made more accurate for heap sizes up to 32G (using CompressedOops), resulting in them dropping by 10-50% in practice. This also results in less number of flushes and compactions due to "fatter" flushes. YMMV. As a result, the actual heap usage of the memstore before being flushed may increase by up to 100%. If configured memory limits for the region server had been tuned based on observed usage, this change could result in worse GC behavior or even OutOfMemory errors. Set the environment property (not hbase-site.xml) "hbase.memorylayout.use.unsafe" to false to disable.


[[upgrade1.4.replication]]
==== Replication peer's TableCFs config

Before 1.4, the table name can't include namespace for replication peer's TableCFs config. It was fixed by add TableCFs to ReplicationPeerConfig which was stored on Zookeeper. So when upgrade to 1.4, you have to update the original ReplicationPeerConfig data on Zookeeper firstly. There are four steps to upgrade when your cluster have a replication peer with TableCFs config.

* Disable the replication peer.
* If master has permission to write replication peer znode, then rolling update master directly. If not, use TableCFsUpdater tool to update the replication peer's config.
[source,bash]
----
$ bin/hbase org.apache.hadoop.hbase.replication.master.TableCFsUpdater update
----
* Rolling update regionservers.
* Enable the replication peer.

Notes:

* Can't use the old client(before 1.4) to change the replication peer's config. Because the client will write config to Zookeeper directly, the old client will miss TableCFs config. And the old client write TableCFs config to the old tablecfs znode, it will not work for new version regionserver.

[[upgrade1.4.rawscan]]
==== Raw scan now ignores TTL

Doing a raw scan will now return results that have expired according to TTL settings.

[[upgrade1.3]]
=== Upgrading from pre-1.3 to 1.3+
If running Integration Tests under Kerberos, see <<upgrade2.0.it.kerberos>>.


[[upgrade1.0]]
=== Upgrading to 1.x

Please consult the documentation published specifically for the version of HBase that you are upgrading to for details on the upgrade process.