Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
(See accompanying source code for licensing information)
This software is created to demonstrate Apache Bigtop for processing big data sets.
The application consists of the following modules
JIRA
issueJIRA
issueYou‘ll need to have version 2.0 of gradle
installed and set-up correctly in order to follow along these instructions. We could have used the gradle-wrapper
to avoid having to install gradle
, but the bigtop
project includes all gradle*
directories in .gitignore
. So, that’s not going to work.
gradle clean build
will build the bigpetstore jar
. The jar
will be located in the build\libs
directory.
gradle clean integrationTest -P ITProfile=pig
gradle clean integrationTest -P ITProfile=mahout
If you don't specify any profile-name, or if you specify an invalid-name for the integrationTest
task, no integration tests will be run.
Note: At this stage, only the Pig
and Mahout
profiles are working. Will continue to update this area as further work is completed.
gradle eclipse
to create an eclipse project.Note whenever you modify the dependencies, you will need to run the gradle eclipse
again. Refresh the project after doing so. You'd also need to have the scala
plugin installed. Also, having a gradle
plugin would be quite useful as well, for ex. when you want to update dependencies.
The bigpetstore project exemplifies the hadoop ecosystem for newcomers, and also for benchmarking and comparing functional space of tools.
The end goal is to run many different implementations of each phase using different tools, thus exemplifying overlap of tools in the hadoop ecosystem, and allowing people to benchmark/compare tools using a common framework and easily understood use case
The first step is to generate a raw data set. This is done by the “GeneratePetStoreTransactionsInputFormat”:
The first MapReduce job in the pipeline runs a simple job which takes this input format and forwards its output. The result is a list of “transactions”. Each transaction is a tuple of the format
{state,name,date,price,product}.
The next phase of the application processes the data to create basic aggregations. For example with both pig and hive these could easily include
Now, say we want to cluster the states, so as to put different states into different buying categories for our marketing team to deal with differently.
Note: For running the code using the hadoop jar
command instead of the gradle
tasks, you will need to set the classpath appropriately. The discussion after this comment in JIRA could also be useful apart from these instructions.
We are going to use a fat-jar in order to avoid specifying the entire classpath ourselves.
The fat-jar is required when we are running the application on a hadoop cluster. The other way would be to specify all the dependencies (including the transitive ones) manually while running the hadoop job. Fat-jars make it easier to bundle almost all the dependencies inside the distribution jar itself.
gradle clean shadowJar -Pfor-cluster
This command will build the fat-jar with all the dependencies bundled in except the hadoop, mahout and pig dependencies, which we'll specify using -libjars
option while running the hadoop job. These dependencies are excluded to avoid conflicts with the jars provided by hadoop itself.
The generated jar will be inside the build/libs
dir, with name like BigPetStore-x.x.x-SNAPSHOT-all.jar
. For the remainig discussion I'll refer to this jar by bps.jar
.
You'll need both mahout and pig jars with the hadoop classes excluded. Commonly, you can find both of these in their respective distros. The required pig jar is generally named like pig-x.x.x-withouthadoop.jar
and the mahout jar would be named like mahout-core-job.jar
. If you want, you can build those yourself by following the instructions in this JIRA comment]. For the remaining discussion, I am going to refer to these two jars by pig-withouthadoop.jar
and mahout-core-job.jar
.
export JARS="/usr/lib/pig/pig-withouthadoop.jar,/usr/lib/mahout/mahout-core-job.jar"
We also need these jars to be present on the client side to kick-off the jobs. Reusing the JARS
variable to put the same jars on the client classpath.
export HADOOP_CLASSPATH=`echo $JARS | sed s/,/:/g`
hadoop jar bps.jar org.apache.bigtop.bigpetstore.generator.BPSGenerator 1000000 bigpetstore/gen
hadoop jar bps.jar org.apache.bigtop.bigpetstore.etl.PigCSVCleaner -libjars $JARS bigpetstore/gen/ bigpetstore/ custom_pigscript.pig
hadoop jar bps.jar org.apache.bigtop.bigpetstore.recommend.ItemRecommender -libjars $JARS bigpetstore/pig/Mahout bigpetstore/Mahout/AlsFactorization bigpetstore/Mahout/AlsRecommendations
... (will add more steps as we add more phases to the workflow) ...
Put the jar in s3. Right now there is a copy of it at the url below.
Download the elastic-mapreduce ruby shell script. create your “credentials.json” file.
Now run this to generate 1,000,000 pet store transactions:
./elastic-mapreduce --create --jar s3://bigpetstore/bigpetstore.jar
--main-class org.apache.bigtop.bigpetstore.generator.BPSGenerator
--num-instances 10
--arg 1000000
--arg s3://bigpetstore/data/generated
--hadoop-version “2.2.0”
--master-instance-type m1.medium
--slave-instance-type m1.medium
...Now lets clean the data with pig...
Replace the above “main-class”, and “--arg” options with --main-class org.apache.bigtop.bigpetstore.etl.PigCSVCleaner --arg s3://bigpetstore/data/generated --arg s3://bigpetstore/data/pig_out (optional, you can send a script referencing the cleaned $input path to do some custom analytics, see the BPS_Analytics.pig script and companion http://jayunit100.github.io/bigpetstore) as an example). --arg s3://path_to_custom_analytics_script.pig
(note about pig: We support custom pig scripts.... for EMR, custom pig scripts will need to point to a local path, so youll have to put that script on the machine as part of EMR setup w/ a custom script).
...
And so on.