Running as a Hadoop client, Apache Kylin's metadata and Cube data are persistended in Hadoop (HBase and HDFS), so the upgrade is relatively easy and user does not need worry about data loss. The upgrade can be performed in the following steps:
$KYLIN_HOME/conf/*) into the new ones. It is not recommended to overwrite the new configuration files, although that works in most cases. If you have modified tomcat configuration ($KYLIN_HOME/tomcat/conf/), do the same for it.bin/kylin.sh stopKYLIN_HOME env variable to the new installation folder. If you have set KYLIN_HOME in ~/.bash_profile or other scripts, remember to update them as well.$KYLIN_HOME/bin/kylin start. After be started, login Kylin web to check whether your cubes can be loaded correctly.Below are versions specific guides:
kylin.server.mode=stream_coordinator for the coordinator node, which can no longer be set to kylin.server.mode=all.kylin.security.encrypt.cipher.ivSpec in kylin 3.1.3. If you uses the default value, there is no need to modify the encryption password in kylin.properties. If you changes the value of kylin.security.encrypt.cipher.ivSpec, the encrypted password needs to be re-encrypted.The encryption algorithm may be used in kylin.metadata.url(mysql password), kylin.security.ldap.connection-password, etc.
1)Set Config on web of kylin v3.1.0 is turned off by default. 2) KYLIN-4478 modified the encryption algorithm. If you enable LDAP in kylin, you need to re-encoded the password of the LDAP server configured in kylin.properties:
cd $KYLIN_HOME/tomcat/webapps/kylin/WEB-INF/lib java -classpath kylin-server-base-\<versioin\>.jar:kylin-core-common-\<versioin\>.jar:spring-beans-4.3.10.RELEASE.jar:spring-core-4.3.10.RELEASE.jar:commons-codec-1.7.jar org.apache.kylin.rest.security.PasswordPlaceholderConfigurer AES <your_password>
Then update the re-encoded password to kylin.security.ldap.connection-password=<your_password_encrypted>.
kylin.source.jdbc.sqoop-home need be the location of sqoop installation, not its “bin” subfolder, please modify it if you're using RDBMS as the data source.Kylin v2.2.0 cube metadata is compitable with v2.1.0, but you need aware the following changes:
Kylin v2.1.0 cube metadata is compitable with v2.0.0, but you need aware the following changes.
ERROR: Legacy ACL metadata detected. Please migrate ACL metadata first. Run command 'bin/kylin.sh org.apache.kylin.tool.AclTableMigrationCLI MIGRATE'.
After the migration finished, you can delete the legacy “kylin_metadata_user” and “kylin_metadata_acl” tables from HBase.
From v2.1, Kylin hides the default settings in “conf/kylin.properties”; You only need uncomment or add the customized properties in it.
Spark is upgraded from v1.6.3 to v2.1.1, if you customized Spark configurations in kylin.properties, please upgrade them as well by referring to Spark documentation.
If you are running Kylin with two clusters (compute/query separated), need copy the big metadata files (which are persisted in HDFS instead of HBase) from the Hadoop cluster to HBase cluster.
hadoop distcp hdfs://compute-cluster:8020/kylin/kylin_metadata/resources hdfs://query-cluster:8020/kylin/kylin_metadata/resources
Kylin v2.0.0 can read v1.6.0 metadata directly. Please follow the common upgrade steps above.
Configuration names in kylin.properties have changed since v2.0.0. While the old property names still work, it is recommended to use the new property names as they follow the naming convention and are easier to understand. There is a mapping from the old properties to the new properties.
Kylin v1.5.4 and v1.6.0 are compatible in metadata. Please follow the common upgrade steps above.
Kylin v1.5.3 and v1.5.4 are compatible in metadata. Please follow the common upgrade steps above.
Kylin v1.5.3 metadata is compatible with v1.5.2, your cubes don't need rebuilt, as usual, some actions need to be performed:
The HBase tables for existing cubes need be updated to the latest coprocessor; Follow this guide to update;
From 1.5.3, Kylin doesn't need Hive to merge small files anymore; For users who copy the conf/ from previous version, please remove the “merge” related properties in kylin_hive_conf.xml, including “hive.merge.mapfiles”, “hive.merge.mapredfiles”, and “hive.merge.size.per.task”; this will save the time on extracting data from Hive.
Kylin v1.5.2 metadata is compatible with v1.5.1, your cubes don't need upgrade, while some actions need to be performed:
The HBase tables for existing cubes need be updated to the latest coprocessor; Follow this guide to update;
In v1.5.2 several properties are deprecated, and several new one are added:
Deprecated:
New:
These new parameters determines how to split HBase region; To use different size you can overwite these params in Cube level.
When copy from old kylin.properties file, suggest to remove the deprecated ones and add the new ones.
A new job conf file named “kylin_job_conf_inmem.xml” is added in “conf” folder; As Kylin 1.5 introduced the “fast cubing” algorithm, which aims to leverage more memory to do the in-mem aggregation; Kylin will use this new conf file for submitting the in-mem cube build job, which requesting different memory with a normal job; Please update it properly according to your cluster capacity.
Besides, if you have used separate config files for different capacity cubes, for example “kylin_job_conf_small.xml”, “kylin_job_conf_medium.xml” and “kylin_job_conf_large.xml”, please note that they are deprecated now; Only “kylin_job_conf.xml” and “kylin_job_conf_inmem.xml” will be used for submitting cube job; If you have cube level job configurations (like using different Yarn job queue), you can customize at cube level, check KYLIN-1706