Building CarbonData

Prerequisites for building CarbonData:

I. Clone CarbonData

$ git clone https://github.com/apache/incubator-carbondata.git

II. Build the project

  • Build without test.By default carbon takes Spark 1.5.2 to build the project
$ mvn -DskipTests clean package
  • Build with different spark versions
$ mvn -DskipTests -Pspark-1.5 -Dspark.version=1.5.0 clean package
$ mvn -DskipTests -Pspark-1.5 -Dspark.version=1.5.1 clean package
$ mvn -DskipTests -Pspark-1.5 -Dspark.version=1.5.2 clean package
 
$ mvn -DskipTests -Pspark-1.6 -Dspark.version=1.6.0 clean package
$ mvn -DskipTests -Pspark-1.6 -Dspark.version=1.6.1 clean package
$ mvn -DskipTests -Pspark-1.6 -Dspark.version=1.6.2 clean package
  • Build the assembly jar which includes Spark and Hadoop jars
$ mvn clean -DskipTests -Pinclude-all package
  • Build with test
$ mvn clean package

Developing CarbonData

The CarbonData committers use IntelliJ IDEA and Eclipse IDE to develop.

IntelliJ IDEA

  • Download IntelliJ at https://www.jetbrains.com/idea/ and install the Scala plug-in for IntelliJ at http://plugins.jetbrains.com/plugin/?id=1347
  • Go to “File -> Import Project”, locate the CarbonData source directory, and select “Maven Project”.
  • In the Import Wizard, select “Import Maven projects automatically” and leave other settings at their default.
  • Leave other settings at their default and you should be able to start your development.
  • When you run the scala test, sometimes you will get out of memory exception. You can increase your VM memory usage by the following setting, for example:
-XX:MaxPermSize=512m -Xmx3072m

You can also make those setting to be the default by setting to the “Defaults -> ScalaTest”.

Eclipse

  • Download the Scala IDE (preferred) or install the scala plugin to Eclipse.
  • Import the CarbonData Maven projects (“File” -> “Import” -> “Maven” -> “Existing Maven Projects” -> locate the CarbonData source directory).