layout: dev40 title: Setup Development Env categories: development permalink: /development40/dev_env.html

Developers want to run Kylin4 test cases or applications at their development machine.

Following this tutorial, you can easily build a kylin4 development environment on your local machine without connecting to a Hadoop client or sandbox.

Environment on the dev machine

Install Maven

Download Maven 3.5.4 and above version: http://maven.apache.org/download.cgi, we create a symbolic so that mvn can be run anywhere.

{% highlight Groff markup %} cd ~ wget http://xenia.sote.hu/ftp/mirrors/www.apache.org/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gz tar -xzvf apache-maven-3.5.4-bin.tar.gz ln -s /root/apache-maven-3.5.4/bin/mvn /usr/bin/mvn {% endhighlight %}

Install Spark

Manually install the Spark binary in a local folder like /usr/local/spark. Kylin4 supports Spark 2.4.7, you need to get the download link from the spark download page.

{% highlight Groff markup %} wget -O /tmp/spark-2.4.7-bin-hadoop2.7.tgz https://archive.apache.org/dist/spark/spark-2.4.7/spark-2.4.7-bin-hadoop2.7.tgz cd /usr/local tar -zxvf /tmp/spark-2.4.7-bin-hadoop2.7.tgz ln -s spark-2.4.7-bin-hadoop2.7 spark {% endhighlight %}

Compile

First clone the Kylin project to your local:

{% highlight Groff markup %} git clone https://github.com/apache/kylin.git {% endhighlight %}

Install Kylin artifacts to the maven repository.

{% highlight Groff markup %} mvn clean install -DskipTests {% endhighlight %}

Run unit tests

Run unit tests to validate basic function of each classes.

{% highlight Groff markup %} mvn clean test {% endhighlight %}

Run integration tests

Executing the following command will run the unit test and integration test. Both unit and integration tests need to be passed before code is submitted.

{% highlight Groff markup %} mvn clean test -DskipRunIt=false {% endhighlight %}

To learn more about test, please refer to How to test.

Launch Kylin Web Server locally

Copy server/src/main/webapp/WEB-INF to webapp/app/WEB-INF

{% highlight Groff markup %} cp -r server/src/main/webapp/WEB-INF webapp/app/WEB-INF {% endhighlight %}

Download JS for Kylin web GUI. npm is part of Node.js, please search about how to install it on your OS.

{% highlight Groff markup %} cd webapp npm install -g bower bower --allow-root install {% endhighlight %}

If you encounter network problem when run “bower install”, you may try:

{% highlight Groff markup %} git config --global url.“git://”.insteadOf https:// {% endhighlight %}

If errors occur during installing Kylin's frontend dependencies due to network latency or some packages not obtainable by default registry, please refer to How to Set up Frontend Registry

Note, if on Windows, after install bower, need to add the path of “bower.cmd” to system environment variable ‘PATH’, and then run:

{% highlight Groff markup %} bower.cmd --allow-root install {% endhighlight %}

Find the following configuration in examples/test_case_data/sandbox/kylin.properties and modify them according to the following example:

# Need to use absolute pat
kylin.metadata.url=${KYLIN_SOURCE_DIR}/examples/test_case_data/sample_local
kylin.storage.url=${KYLIN_SOURCE_DIR}/examples/test_case_data/sample_local
kylin.env.zookeeper-is-local=true
kylin.env.hdfs-working-dir=file://$KYLIN_SOURCE_DIR/examples/test_case_data/sample_local
kylin.engine.spark-conf.spark.master=local
# Need to create `/path/to/local/dir` manually
kylin.engine.spark-conf.spark.eventLog.dir=/path/to/local/dir
kylin.engine.spark-conf.spark.sql.shuffle.partitions=1
kylin.env=LOCAL

In IDE, launch org.apache.kylin.rest.DebugTomcat. Please set the path of “server” module as the “Working directory”, set “kylin-server” for “Use classpath of module”, and check “Include dependencies with ‘Provided’ scope” option in IntelliJ IDEA 2018. If you're using IntelliJ IDEA 2017 and older, you need modify “server/kylin-server.iml” file, replace all “PROVIDED” to “COMPILE”, otherwise an “java.lang.NoClassDefFoundError: org/apache/catalina/LifecycleListener” error may be thrown..

And adjust VM options:

-Dspark.local=true

DebugTomcat Config

By default Kylin server will listen on 7070 port; If you want to use another port, please specify it as a parameter when run DebugTomcat.

Check Kylin Web at http://localhost:7070/kylin (user:ADMIN, password:KYLIN)

Setup IDE code formatter

In case you're writting code for Kylin, you should make sure that your code in expected formats.

For Eclipse users, just format the code before committing the code.

For intellij IDEA users, you have to do a few more steps:

  1. Install “Eclipse Code Formatter” and use “org.eclipse.jdt.core.prefs” and “org.eclipse.jdt.ui.prefs” in core-common/.settings to configure “Eclipse Java Formatter config file” and “Import order”

    Eclipse_Code_Formatter_Config

  2. Go to Preference => Code Style => Java, set “Scheme” to Default, and set both “Class count to use import with ‘*’” and “Names count to use static import with ‘*’” to 99.

    Kylin_Intellj_Code_Style

  3. Disable intellij IDEA's “Optimize imports on the fly”

    Disable_Optimize_On_The_Fly

  4. Format the code before committing the code.

Setup IDE license header template

Each source file should include the following Apache License header {% highlight Groff markup %} Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

 http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. {% endhighlight %}

The checkstyle plugin will check the header rule when packaging also. The license file locates under dev-support/checkstyle-apache-header.txt. To make it easy for developers, please add the header as Copyright Profile and set it as default for Kylin project. Apache License Profile