Initial implementation of Apache Fluo YARN launcher (#1)
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..ccc651c
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,8 @@
+.classpath
+.project
+.settings
+target
+logs
+*.iml
+.idea
+.DS_Store
diff --git a/README.md b/README.md
index ac14fd2..25b3a3e 100644
--- a/README.md
+++ b/README.md
@@ -18,3 +18,93 @@
# Fluo YARN launcher
A tool for running Apache Fluo applications in Hadoop YARN.
+
+## Requirements
+
+The Fluo YARN launcher requires [Apache Fluo][Fluo] and [Hadoop YARN][YARN]:
+
+| Software | Recommended Version | Minimum Version |
+|-------------|---------------------|-----------------|
+| [Fluo] | 1.2.0 | 1.2.0 |
+| [YARN] | 2.7.2 | 2.6.0 |
+
+See the [related projects page][related] for external projects that may help in setting up these dependencies.
+
+## Set up your Fluo application
+
+Before you can launch a Fluo application in YARN, you should follow Fluo's [install instructions][install]
+to install Fluo and initialize a Fluo application. After your application has been initialized, follow
+the insructions below to install the Fluo YARN launcher and run your application in YARN. Avoid using the
+`fluo` command to start local oracle and worker processes if you are running in YARN.
+
+## Install and Configure Fluo YARN launcher
+
+Before you can install the Fluo YARN launcher, you will need to obtain a distribution tarball. It is
+recommended that you download the [latest release][release]. You can also build a distribution from the
+master branch by following these steps which create a tarball in `distribution/target`:
+
+ git clone https://github.com/apache/fluo-yarn.git
+ cd fluo-yarn/
+ mvn package
+
+After you obtain a Fluo YARN distribution tarball, follow these steps to install Fluo.
+
+1. Choose a directory with plenty of space, untar the distribution, and run `fetch.sh` to retrieve dependencies:
+
+ tar -xvzf fluo-yarn-1.0.0-bin.tar.gz
+ cd fluo-yarn-1.0.0
+ ./lib/fetch.sh
+
+ The distribution contains a `fluo-yarn` script in `bin/` that administers Fluo and the
+ following configuration files in `conf/`:
+
+ | Configuration file | Description |
+ |-----------------------------|-------------------------------------------------------------------------|
+ | [fluo-yarn-env.sh] | Configures classpath for `fluo-yarn` script. Required for all commands. |
+ | [fluo-yarn.properties] | Configures how application runs in YARN. Required for `start` command. |
+ | [log4j.properties] | Configures logging |
+
+2. Configure [fluo-yarn-env.sh]
+
+ * Set `FLUO_HOME` if it is not in your environment
+ * Modify `FLUO_CONN_PROPS` if you don't want use the default.
+
+3. Configure [fluo-yarn.properties] to set how your application will be launched in YARN:
+
+ * YARN resource manager hostname
+ * Number of oracle and worker instances
+ * Max memory usage per oracle/worker
+
+ If you are managing multiple Fluo applications in YARN, you can copy this file and configure it for
+ each application.
+
+## Start Fluo application in YARN
+
+Follow the instructions below to start a Fluo application which contain an oracle and multiple workers.
+
+1. Configure [fluo-yarn-env.sh] and [fluo-yarn.properties] if you have not already.
+
+2. Run the commands below to start your Fluo application in YARN.
+
+ fluo-yarn start myapp conf/fluo-yarn.properties
+
+ The commands will retrieve your application configuration and observer jars (using your
+ application name) before starting the application in YARN.
+
+## Manage Fluo application in YARN
+
+Except for stopping your application in YARN, the `fluo` script can be used to manage your application using the
+`scan` and `wait` commands. See Fluo's [install instruction][install] for more information.
+
+When you want you stop your Fluo application, use the the YARN resource manager or the
+`yarn application -kill <App ID>` to stop the application in YARN.
+
+[Fluo]: https://fluo.apache.org/
+[YARN]: http://hadoop.apache.org/
+[related]: https://fluo.apache.org/related-projects/
+[related]: https://fluo.apache.org/related-projects/
+[install]: https://github.com/apache/incubator-fluo/blob/master/docs/install.md
+[release]: https://fluo.apache.org/download/
+[fluo-yarn-env.sh]: distribution/conf/fluo-yarn-env.sh
+[fluo-yarn.properties]: distribution/conf/fluo-yarn.properties
+[log4j.properties]: distribution/conf/log4j.properties
diff --git a/core/pom.xml b/core/pom.xml
new file mode 100644
index 0000000..7b6f455
--- /dev/null
+++ b/core/pom.xml
@@ -0,0 +1,68 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ agreements. See the NOTICE file distributed with this work for additional information regarding
+ copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance with the License. You may obtain a
+ copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software distributed under the License
+ is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ or implied. See the License for the specific language governing permissions and limitations under
+ the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.fluo</groupId>
+ <artifactId>fluo-yarn-project</artifactId>
+ <version>1.0.0-SNAPSHOT</version>
+ <relativePath>../pom.xml</relativePath>
+ </parent>
+ <artifactId>fluo-yarn-core</artifactId>
+ <name>Apache Fluo YARN Core</name>
+ <description>This module contains all code necessary to run Apache Fluo on a YARN cluster using Apache Twill</description>
+ <dependencies>
+ <dependency>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.fluo</groupId>
+ <artifactId>fluo-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.fluo</groupId>
+ <artifactId>fluo-core</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ <exclusions>
+ <!-- Excluded so we use asm jar included by twill-yarn. See FLUO-409 -->
+ <exclusion>
+ <groupId>asm</groupId>
+ <artifactId>asm</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.twill</groupId>
+ <artifactId>twill-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.twill</groupId>
+ <artifactId>twill-ext</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.twill</groupId>
+ <artifactId>twill-yarn</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-api</artifactId>
+ </dependency>
+ </dependencies>
+</project>
diff --git a/core/src/main/java/org/apache/fluo/yarn/core/FluoYarnEnv.java b/core/src/main/java/org/apache/fluo/yarn/core/FluoYarnEnv.java
new file mode 100644
index 0000000..dcf2041
--- /dev/null
+++ b/core/src/main/java/org/apache/fluo/yarn/core/FluoYarnEnv.java
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.fluo.yarn.core;
+
+import java.io.File;
+import java.nio.file.Paths;
+import java.util.Objects;
+import java.util.Properties;
+
+import com.google.common.base.Preconditions;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+
+public class FluoYarnEnv {
+
+ private Properties props;
+ private String applicationName;
+ private String connPropsPath;
+ private String logPropsPath;
+ private String bundledJarPath = null;
+ private YarnConfiguration yarnConfig = null;
+
+ public FluoYarnEnv(String yarnProps, String connProps, String logProps, String appName,
+ String jarPath) {
+ Objects.requireNonNull(yarnProps);
+ Objects.requireNonNull(connProps);
+ Objects.requireNonNull(logProps);
+ Objects.requireNonNull(appName);
+ Objects.requireNonNull(jarPath);
+ verifyPath(connProps);
+ verifyPath(yarnProps);
+ verifyPath(logProps);
+ verifyPath(jarPath);
+ props = FluoYarnProperties.loadFromFile(yarnProps);
+ connPropsPath = connProps;
+ logPropsPath = logProps;
+ applicationName = appName;
+ bundledJarPath = jarPath;
+
+ }
+
+ public String getBundledJarPath() {
+ return bundledJarPath;
+ }
+
+ public String getBundledJarName() {
+ return Paths.get(getBundledJarPath()).getFileName().toString();
+ }
+
+ public String getLogPropsPath() {
+ return logPropsPath;
+ }
+
+ public String getConnPropsPath() {
+ return connPropsPath;
+ }
+
+ public String getApplicationName() {
+ return applicationName;
+ }
+
+ private static void verifyPath(String path) {
+ File f = new File(path);
+ Preconditions.checkState(f.exists());
+ Preconditions.checkState(f.canRead());
+ }
+
+ public YarnConfiguration getYarnConfiguration() {
+ if (yarnConfig == null) {
+ yarnConfig = new YarnConfiguration();
+ yarnConfig.set("mapreduce.framework.name", "yarn");
+ yarnConfig.set("yarn.resourcemanager.hostname", getYarnResourceManager());
+ }
+ return yarnConfig;
+ }
+
+ public String getYarnResourceManager() {
+ return props.getProperty(FluoYarnProperties.YARN_RESOURCE_MANAGER_PROP,
+ FluoYarnProperties.YARN_RESOURCE_MANAGER_DEFAULT);
+ }
+
+ public String getZookeepers() {
+ return props.getProperty(FluoYarnProperties.ZOOKEEPERS_PROP,
+ FluoYarnProperties.ZOOKEEPERS_DEFAULT);
+ }
+
+ public int getWorkerCores() {
+ return Integer.valueOf(props.getProperty(FluoYarnProperties.WORKER_NUM_CORES_PROP,
+ FluoYarnProperties.WORKER_NUM_CORES_DEFAULT));
+ }
+
+ public int getWorkerInstances() {
+ return Integer.valueOf(props.getProperty(FluoYarnProperties.WORKER_INSTANCES_PROP,
+ FluoYarnProperties.WORKER_INSTANCES_DEFAULT));
+ }
+
+ public int getWorkerMaxMemory() {
+ return Integer.valueOf(props.getProperty(FluoYarnProperties.WORKER_MAX_MEMORY_MB_PROP,
+ FluoYarnProperties.WORKER_MAX_MEMORY_MB_DEFAULT));
+ }
+
+ public int getOracleCores() {
+ return Integer.valueOf(props.getProperty(FluoYarnProperties.ORACLE_NUM_CORES_PROP,
+ FluoYarnProperties.ORACLE_NUM_CORES_DEFAULT));
+ }
+
+ public int getOracleInstances() {
+ return Integer.valueOf(props.getProperty(FluoYarnProperties.ORACLE_INSTANCES_PROP,
+ FluoYarnProperties.ORACLE_INSTANCES_DEFAULT));
+ }
+
+ public int getOracleMaxMemory() {
+ return Integer.valueOf(props.getProperty(FluoYarnProperties.ORACLE_MAX_MEMORY_MB_PROP,
+ FluoYarnProperties.ORACLE_MAX_MEMORY_MB_DEFAULT));
+ }
+
+ public int getTotalInstances() {
+ return getOracleInstances() + getWorkerInstances();
+ }
+}
diff --git a/core/src/main/java/org/apache/fluo/yarn/core/FluoYarnLauncher.java b/core/src/main/java/org/apache/fluo/yarn/core/FluoYarnLauncher.java
new file mode 100644
index 0000000..7da612e
--- /dev/null
+++ b/core/src/main/java/org/apache/fluo/yarn/core/FluoYarnLauncher.java
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.fluo.yarn.core;
+
+import java.io.File;
+import java.util.Collection;
+
+import org.apache.twill.api.ResourceReport;
+import org.apache.twill.api.ResourceSpecification;
+import org.apache.twill.api.TwillApplication;
+import org.apache.twill.api.TwillController;
+import org.apache.twill.api.TwillPreparer;
+import org.apache.twill.api.TwillRunResources;
+import org.apache.twill.api.TwillRunnerService;
+import org.apache.twill.api.TwillSpecification;
+import org.apache.twill.ext.BundledJarRunnable;
+import org.apache.twill.ext.BundledJarRunner;
+import org.apache.twill.yarn.YarnTwillRunnerService;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class FluoYarnLauncher {
+
+ private static final Logger log = LoggerFactory.getLogger(FluoYarnLauncher.class);
+
+ private static final String ORACLE_ID = "Oracle";
+ private static final String WORKER_ID = "Worker";
+ private static final String CONN_PROPS = "fluo-conn.properties";
+ private static final String LOG4J_PROPS = "log4j.properties";
+
+ private static class FluoYarnApp implements TwillApplication {
+
+ private FluoYarnEnv env;
+
+ FluoYarnApp(FluoYarnEnv env) {
+ this.env = env;
+ }
+
+ @Override
+ public TwillSpecification configure() {
+ ResourceSpecification oracleResources =
+ ResourceSpecification.Builder.with().setVirtualCores(env.getOracleCores())
+ .setMemory(env.getOracleMaxMemory(), ResourceSpecification.SizeUnit.MEGA)
+ .setInstances(env.getOracleInstances()).build();
+
+ ResourceSpecification workerResources =
+ ResourceSpecification.Builder.with().setVirtualCores(env.getWorkerCores())
+ .setMemory(env.getWorkerMaxMemory(), ResourceSpecification.SizeUnit.MEGA)
+ .setInstances(env.getWorkerInstances()).build();
+
+ return TwillSpecification.Builder.with().setName("fluo-app-" + env.getApplicationName())
+ .withRunnable().add(ORACLE_ID, new BundledJarRunnable(), oracleResources)
+ .withLocalFiles().add(env.getBundledJarName(), new File(env.getBundledJarPath()), false)
+ .add(CONN_PROPS, new File(env.getConnPropsPath()), false)
+ .add(LOG4J_PROPS, new File(env.getLogPropsPath()), false).apply()
+ .add(WORKER_ID, new BundledJarRunnable(), workerResources).withLocalFiles()
+ .add(env.getBundledJarName(), new File(env.getBundledJarPath()), false)
+ .add(CONN_PROPS, new File(env.getConnPropsPath()), false)
+ .add(LOG4J_PROPS, new File(env.getLogPropsPath()), false).apply().anyOrder().build();
+ }
+ }
+
+ private static int getNumRunning(TwillController controller) {
+ ResourceReport report = controller.getResourceReport();
+ if (report == null) {
+ return 0;
+ }
+ int total = 0;
+ Collection<TwillRunResources> resources = report.getRunnableResources(ORACLE_ID);
+ if (resources != null) {
+ total += resources.size();
+ }
+ resources = report.getRunnableResources(WORKER_ID);
+ if (resources != null) {
+ total += resources.size();
+ }
+ return total;
+ }
+
+ public static void main(String[] args) throws Exception {
+
+ if (args.length != 5) {
+ System.err.println("Invalid arguments");
+ System.exit(-1);
+ }
+
+ String connProps = args[0];
+ String yarnProps = args[1];
+ String logProps = args[2];
+ String appName = args[3];
+ String jarPath = args[4];
+
+ FluoYarnEnv env = new FluoYarnEnv(yarnProps, connProps, logProps, appName, jarPath);
+
+ BundledJarRunner.Arguments oracleArgs =
+ new BundledJarRunner.Arguments.Builder().setJarFileName(env.getBundledJarName())
+ .setLibFolder("lib").setMainClassName("org.apache.fluo.command.FluoOracle")
+ .setMainArgs(new String[] {CONN_PROPS, appName}).createArguments();
+
+ BundledJarRunner.Arguments workerArgs =
+ new BundledJarRunner.Arguments.Builder().setJarFileName(env.getBundledJarName())
+ .setLibFolder("lib").setMainClassName("org.apache.fluo.command.FluoWorker")
+ .setMainArgs(new String[] {CONN_PROPS, appName}).createArguments();
+
+ TwillRunnerService twillRunner =
+ new YarnTwillRunnerService(env.getYarnConfiguration(), env.getZookeepers());
+ twillRunner.start();
+
+ TwillPreparer preparer =
+ twillRunner.prepare(new FluoYarnApp(env))
+ .addJVMOptions("-Dlog4j.configuration=file:$PWD/" + LOG4J_PROPS)
+ .withArguments(ORACLE_ID, oracleArgs.toArray())
+ .withArguments(WORKER_ID, workerArgs.toArray());
+
+ TwillController controller = preparer.start();
+
+ ResourceReport report = controller.getResourceReport();
+ log.info("Waiting for Fluo application '{}' to start in YARN...", appName);
+ while (report == null) {
+ Thread.sleep(500);
+ report = controller.getResourceReport();
+ }
+ String appID = report.getApplicationId();
+ log.info("Fluo application '{}' has started in YARN with ID '{}'", appName, appID);
+
+ log.info("Waiting for all containers of Fluo application '{}' to start in YARN...", appName);
+ int numRunning = getNumRunning(controller);
+ while (numRunning != env.getTotalInstances()) {
+ log.info("{} of {} containers have started in YARN", numRunning, env.getTotalInstances());
+ Thread.sleep(2000);
+ numRunning = getNumRunning(controller);
+ }
+ log.info("{} of {} containers have started in YARN", numRunning, env.getTotalInstances());
+ log.info("Fluo application '{}' has successfully started in YARN with ID '{}'", appName, appID);
+ }
+}
diff --git a/core/src/main/java/org/apache/fluo/yarn/core/FluoYarnProperties.java b/core/src/main/java/org/apache/fluo/yarn/core/FluoYarnProperties.java
new file mode 100644
index 0000000..4d34647
--- /dev/null
+++ b/core/src/main/java/org/apache/fluo/yarn/core/FluoYarnProperties.java
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.fluo.yarn.core;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.util.Properties;
+
+public class FluoYarnProperties {
+
+ private static final String YARN_PREFIX = "fluo.yarn";
+
+ // General properties
+ public static final String YARN_RESOURCE_MANAGER_PROP = YARN_PREFIX + ".resource.manager";
+ public static final String ZOOKEEPERS_PROP = YARN_PREFIX + ".zookeepers";
+ public static final String YARN_RESOURCE_MANAGER_DEFAULT = "localhost";
+ public static final String ZOOKEEPERS_DEFAULT = "localhost/fluo-yarn";
+
+ // Worker properties
+ public static final String WORKER_INSTANCES_PROP = YARN_PREFIX + ".worker.instances";
+ public static final String WORKER_MAX_MEMORY_MB_PROP = YARN_PREFIX + ".worker.max.memory.mb";
+ public static final String WORKER_NUM_CORES_PROP = YARN_PREFIX + ".worker.num.cores";
+ public static final String WORKER_INSTANCES_DEFAULT = "1";
+ public static final String WORKER_MAX_MEMORY_MB_DEFAULT = "1024";
+ public static final String WORKER_NUM_CORES_DEFAULT = "1";
+
+ // Oracle properties
+ public static final String ORACLE_INSTANCES_PROP = YARN_PREFIX + ".oracle.instances";
+ public static final String ORACLE_MAX_MEMORY_MB_PROP = YARN_PREFIX + ".oracle.max.memory.mb";
+ public static final String ORACLE_NUM_CORES_PROP = YARN_PREFIX + ".oracle.num.cores";
+ public static final String ORACLE_INSTANCES_DEFAULT = "1";
+ public static final String ORACLE_MAX_MEMORY_MB_DEFAULT = "512";
+ public static final String ORACLE_NUM_CORES_DEFAULT = "1";
+
+ public static Properties loadFromFile(String propsFilePath) {
+ try {
+ return loadFromStream(new FileInputStream(propsFilePath));
+ } catch (IOException e) {
+ throw new IllegalStateException(e);
+ }
+ }
+
+ public static Properties loadFromStream(FileInputStream fis) {
+ Properties props = new Properties();
+ try {
+ props.load(fis);
+ fis.close();
+ } catch (IOException e) {
+ throw new IllegalStateException(e);
+ }
+ return props;
+ }
+}
diff --git a/distribution/bin/fluo-yarn b/distribution/bin/fluo-yarn
new file mode 100755
index 0000000..367dec4
--- /dev/null
+++ b/distribution/bin/fluo-yarn
@@ -0,0 +1,102 @@
+#! /usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+# agreements. See the NOTICE file distributed with this work for additional information regarding
+# copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance with the License. You may obtain a
+# copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+# or implied. See the License for the specific language governing permissions and limitations under
+# the License.
+
+SOURCE="${BASH_SOURCE[0]}"
+while [ -h "$SOURCE" ]; do
+ bin="$( cd -P "$( dirname "$SOURCE" )" && pwd )"
+ SOURCE="$(readlink "$SOURCE")"
+ [[ $SOURCE != /* ]] && SOURCE="$bin/$SOURCE"
+done
+# Set up variables needed by fluo-env.sh
+export bin="$( cd -P "$( dirname "$SOURCE" )" && pwd )"
+export basedir="$( cd -P ${bin}/.. && pwd )"
+export conf="$basedir/conf"
+export lib="$basedir/lib"
+export cmd="$1"
+case "$cmd" in
+ start) export app="$2" ;;
+esac
+export FLUO_YARN_VERSION=${project.version}
+
+if [ ! -f "$conf/fluo-yarn-env.sh" ]; then
+ echo "fluo-yarn-env.sh must exist in $conf"
+ exit 1
+fi
+source "$conf/fluo-yarn-env.sh"
+export CLASSPATH=$LAUNCHER_CLASSPATH
+
+# stop if any command fails
+set -e
+
+function print_usage {
+ echo -e "Usage: fluo-yarn <command> (<argument> ...)\n"
+ echo -e "Possible commands:\n"
+ echo " classpath Prints the classpath setup by fluo-yarn-env.sh"
+ echo " start <app> <yarnProps> Starts Fluo <app> in YARN using <yarnProps>"
+ echo " version Prints the version of Fluo YARN launcher"
+ echo " "
+ exit 1
+}
+
+function build_bundled_jar {
+ app_dir=$lib/apps/$app
+ mkdir -p "$app_dir/lib"
+ $FLUO_HOME/bin/fluo get-jars "$app" "$app_dir/lib"
+
+ fluo_classpath=$FLUO_CLASSPATH
+
+ IFS=':' read -ra ADDR <<< "$fluo_classpath"
+ for cpentry in "${ADDR[@]}"; do
+ if [[ $cpentry = *\* ]]; then
+ for f in $cpentry; do
+ if [[ -f $f ]]; then
+ cp "$f" "$app_dir/lib/"
+ fi
+ done
+ else
+ cp "$cpentry" "$app_dir/lib/"
+ fi
+ done
+
+ bundled_jar=fluo-app-${app}.jar
+
+ pushd "$app_dir" > /dev/null
+ jar cf $bundled_jar lib
+ popd > /dev/null
+}
+
+case "$1" in
+start)
+ if [ ! -f "$FLUO_CONN_PROPS" ]; then
+ echo "Fluo connection properties file '$FLUO_CONN_PROPS' does not exist"
+ exit 1
+ fi
+ if [ ! -f "$3" ]; then
+ echo "The file argument given for <yarnProps> does not exist at $3"
+ print_usage
+ exit 1
+ fi
+ build_bundled_jar
+ java org.apache.fluo.yarn.core.FluoYarnLauncher "$FLUO_CONN_PROPS" "$3" "$conf/log4j.properties" "$app" "$app_dir/$bundled_jar"
+ ;;
+classpath)
+ echo "$CLASSPATH"
+ ;;
+version)
+ echo "$FLUO_YARN_VERSION"
+ ;;
+*)
+ print_usage
+esac
diff --git a/distribution/conf/fluo-yarn-env.sh b/distribution/conf/fluo-yarn-env.sh
new file mode 100755
index 0000000..6b2f4f3
--- /dev/null
+++ b/distribution/conf/fluo-yarn-env.sh
@@ -0,0 +1,73 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+# agreements. See the NOTICE file distributed with this work for additional information regarding
+# copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with the License. You may
+# obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+# or implied. See the License for the specific language governing permissions and limitations under
+# the License.
+
+## Before fluo-yarn-env.sh is loaded, these environment variables are set and can be used in this file:
+
+# cmd - Command that is being called such as oracle, worker, etc.
+# app - Fluo application name
+# basedir - Root of Fluo YARN launcher installation
+# conf - Directory containing Fluo YARN launcher configuration
+# lib - Directory containing Fluo YARN launcher libraries
+
+####################################
+# General variables that must be set
+####################################
+
+## Fluo installation
+export FLUO_HOME="${FLUO_HOME:-/path/to/fluo}"
+## Hadoop installation
+export HADOOP_PREFIX="${HADOOP_PREFIX:-/path/to/hadoop}"
+## Zookeeper installation
+export ZOOKEEPER_HOME="${ZOOKEEPER_HOME:-/path/to/zookeeper}"
+## File that contains properties need to connect to Fluo
+export FLUO_CONN_PROPS=${FLUO_CONN_PROPS:-$FLUO_HOME/conf/fluo-conn.properties}
+
+###########################
+# Build classpath variables
+###########################
+
+## A classpath needs to built for the YARN launcher and the launched Fluo application
+
+## Classpath for launched Fluo application
+export FLUO_CLASSPATH=$($FLUO_HOME/bin/fluo classpath)
+
+## Classpath of Fluo YARN Launcher
+addToClasspath()
+{
+ local dir=$1
+ local filterRegex=$2
+
+ if [ ! -d "$dir" ]; then
+ echo "ERROR $dir does not exist or not a directory"
+ exit 1
+ fi
+
+ for jar in $dir/*.jar; do
+ if ! [[ $jar =~ $filterRegex ]]; then
+ LAUNCHER_CLASSPATH="$LAUNCHER_CLASSPATH:$jar"
+ fi
+ done
+}
+
+# Any jars matching this pattern is excluded from classpath
+EXCLUDE_RE="(.*log4j.*)|(.*asm.*)|(.*guava.*)|(.*gson.*)"
+LAUNCHER_CLASSPATH="$conf:$lib/*"
+addToClasspath "$ZOOKEEPER_HOME" $EXCLUDE_RE
+addToClasspath "$ZOOKEEPER_HOME/lib" $EXCLUDE_RE
+addToClasspath "$HADOOP_PREFIX/share/hadoop/common" $EXCLUDE_RE
+addToClasspath "$HADOOP_PREFIX/share/hadoop/common/lib" $EXCLUDE_RE
+addToClasspath "$HADOOP_PREFIX/share/hadoop/hdfs" $EXCLUDE_RE
+addToClasspath "$HADOOP_PREFIX/share/hadoop/hdfs/lib" $EXCLUDE_RE
+addToClasspath "$HADOOP_PREFIX/share/hadoop/yarn" $EXCLUDE_RE
+addToClasspath "$HADOOP_PREFIX/share/hadoop/yarn/lib" $EXCLUDE_RE
+export LAUNCHER_CLASSPATH
diff --git a/distribution/conf/fluo-yarn.properties b/distribution/conf/fluo-yarn.properties
new file mode 100644
index 0000000..996b132
--- /dev/null
+++ b/distribution/conf/fluo-yarn.properties
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+# agreements. See the NOTICE file distributed with this work for additional information regarding
+# copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with the License. You may
+# obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+# or implied. See the License for the specific language governing permissions and limitations under
+# the License.
+
+########################
+# Fluo YARN properties
+########################
+
+# NOTE - All properties that have a default are set with it. Uncomment
+# a property if you want to use a value different than the default.
+# Properties that have no default are uncommented and must be set by
+# the user.
+
+## General properties
+## ------------------
+## Zookeeper connection string specifying host and chroot where Fluo YARN launcher stores data.
+## A chroot directory suffix (with max depth of 1) must be specified but doesn't need to be named '/fluo-yarn'.
+#fluo.yarn.zookeepers=localhost/fluo-yarn
+## YARN resource manager hostname. Should match 'yarn.resourcemanager.hostname' property in Hadoop's yarn-site.xml
+#fluo.yarn.resource.manager=localhost
+
+## Oracle properties
+## -----------------
+## Number of oracle yarn instances
+#fluo.yarn.oracle.instances=1
+## Max memory of Oracle yarn containers (in MB)
+#fluo.yarn.oracle.max.memory.mb=512
+## Number of Oracle virtual cores
+#fluo.yarn.oracle.num.cores=1
+
+## Worker properties
+## -----------------
+## Number of Worker instances
+#fluo.yarn.worker.instances=1
+## Max memory of worker YARN containers (in MB). If YARN is killing worker processes consider
+## increasing twill.java.reserved.memory.mb (which defaults to 200 and is set in yarn-site.xml).
+## The twill.java.reserved.memory.mb config determines the gap between the YARN memory limit set
+## below and the java -Xmx setting. For example, if max memory is 1024 and twill reserved memory
+## is 200, the java -Xmx setting will be 1024-200 = 824 MB.
+#fluo.yarn.worker.max.memory.mb=1024
+## Number of worker virtual cores
+#fluo.yarn.worker.num.cores=1
diff --git a/distribution/conf/log4j.properties b/distribution/conf/log4j.properties
new file mode 100644
index 0000000..e140fbf
--- /dev/null
+++ b/distribution/conf/log4j.properties
@@ -0,0 +1,25 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+## Log4j 1.2 file that configures logging for Fluo processes in YARN
+
+log4j.appender.console=org.apache.log4j.ConsoleAppender
+log4j.appender.console.layout=org.apache.log4j.PatternLayout
+log4j.appender.console.layout.ConversionPattern=%d{ISO8601} [%-8c{2}] %-5p: %m%n
+
+log4j.logger.org.apache.curator=ERROR
+log4j.logger.org.apache.zookeeper=ERROR
+
+log4j.rootLogger=INFO, console
diff --git a/distribution/conf/logback.xml b/distribution/conf/logback.xml
new file mode 100644
index 0000000..55d3ccf
--- /dev/null
+++ b/distribution/conf/logback.xml
@@ -0,0 +1,35 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ agreements. See the NOTICE file distributed with this work for additional information regarding
+ copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance with the License. You may obtain a
+ copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software distributed under the License
+ is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ or implied. See the License for the specific language governing permissions and limitations under
+ the License.
+-->
+
+<!-- Configures logging for fluo-yarn command -->
+<configuration>
+ <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
+ <encoder>
+ <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
+ </encoder>
+ </appender>
+
+ <logger name="org.apache.fluo" level="debug"/>
+ <logger name="org.apache.curator" level="error"/>
+ <logger name="org.apache.hadoop" level="info"/>
+ <logger name="org.apache.hadoop.yarn.client.RMProxy" level="error"/>
+ <logger name="org.apache.twill" level="info"/>
+ <logger name="org.apache.twill.yarn.YarnTwillRunnerService" level="error"/>
+ <logger name="org.apache.zookeeper" level="error"/>
+
+ <root level="debug">
+ <appender-ref ref="STDOUT" />
+ </root>
+</configuration>
diff --git a/distribution/lib/fetch.sh b/distribution/lib/fetch.sh
new file mode 100755
index 0000000..0881187
--- /dev/null
+++ b/distribution/lib/fetch.sh
@@ -0,0 +1,83 @@
+#! /usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+# agreements. See the NOTICE file distributed with this work for additional information regarding
+# copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance with the License. You may obtain a
+# copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+# or implied. See the License for the specific language governing permissions and limitations under
+# the License.
+
+lib_dir=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
+maven_prefix=https://repo1.maven.org/maven2
+
+function download {
+ IFS=':' read -ra DEP <<< "$1"
+ dir=$lib_dir/
+ if [ -n "$2" ]; then
+ dir=$lib_dir/$2
+ if [ ! -d $dir ]; then
+ mkdir $dir
+ fi
+ fi
+ group=${DEP[0]}
+ artifact=${DEP[1]}
+ ftype=${DEP[2]}
+ version=${DEP[3]}
+ fn=$artifact-$version.$ftype
+ path="${group//.//}/$artifact/$version/$fn"
+ download_url=$maven_prefix/$path
+
+ if [ -f $dir/$fn ]; then
+ echo "SUCCESS: Dependency exists - $dir/$fn"
+ else
+ wget -q $download_url -P $dir
+ if [ $? == 0 ]; then
+ echo "SUCCESS: Dependency downloaded from $download_url"
+ else
+ echo "ERROR: Dependency download failed - $download_url"
+ fi
+ fi
+}
+
+echo "Fetching Fluo YARN launcher dependencies"
+download aopalliance:aopalliance:jar:1.0
+download ch.qos.logback:logback-classic:jar:1.1.3
+download ch.qos.logback:logback-core:jar:1.1.3
+download com.101tec:zkclient:jar:0.3
+download com.google.code.findbugs:jsr305:jar:2.0.1
+download com.google.code.gson:gson:jar:2.8.0
+download com.google.guava:guava:jar:13.0.1
+download com.yammer.metrics:metrics-annotation:jar:2.2.0
+download com.yammer.metrics:metrics-core:jar:2.2.0
+download net.sf.jopt-simple:jopt-simple:jar:3.2
+download org.apache.kafka:kafka_2.10:jar:0.8.0
+download org.apache.twill:twill-api:jar:0.11.0
+download org.apache.twill:twill-common:jar:0.11.0
+download org.apache.twill:twill-core:jar:0.11.0
+download org.apache.twill:twill-discovery-api:jar:0.11.0
+download org.apache.twill:twill-discovery-core:jar:0.11.0
+download org.apache.twill:twill-ext:jar:0.11.0
+download org.apache.twill:twill-yarn:jar:0.11.0
+download org.apache.twill:twill-zookeeper:jar:0.11.0
+download org.ow2.asm:asm-all:jar:5.0.2
+download org.scala-lang:scala-compiler:jar:2.10.1
+download org.scala-lang:scala-library:jar:2.10.1
+download org.scala-lang:scala-reflect:jar:2.10.1
+download org.slf4j:jcl-over-slf4j:jar:1.7.2
+download org.slf4j:log4j-over-slf4j:jar:1.7.12
+download org.slf4j:slf4j-api:jar:1.7.12
+download org.xerial.snappy:snappy-java:jar:1.0.5
+# See https://github.com/apache/fluo/issues/820
+download io.netty:netty:jar:3.9.9.Final
+
+echo -e "Done!\n"
+echo "NOTE - The dependencies downloaded have been tested with some versions of Fluo, Hadoop, Zookeeper, and Accumulo."
+echo "There is no guarantee they will work with all versions. The Fluo YARN launcher chose to defer dependency resolution to as"
+echo "late as possible in order to make it easier to resolve dependency conflicts. If you run into a dependency"
+echo "conflict in your environment, please consider bringing it up on the Fluo dev list."
diff --git a/distribution/pom.xml b/distribution/pom.xml
new file mode 100644
index 0000000..68e11e9
--- /dev/null
+++ b/distribution/pom.xml
@@ -0,0 +1,90 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ agreements. See the NOTICE file distributed with this work for additional information regarding
+ copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance with the License. You may obtain a
+ copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software distributed under the License
+ is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ or implied. See the License for the specific language governing permissions and limitations under
+ the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.fluo</groupId>
+ <artifactId>fluo-yarn-project</artifactId>
+ <version>1.0.0-SNAPSHOT</version>
+ <relativePath>../pom.xml</relativePath>
+ </parent>
+ <artifactId>fluo-yarn</artifactId>
+ <packaging>pom</packaging>
+ <name>Apache Fluo YARN distribution</name>
+ <description>This module produces a tarball distribution of the YARN launcher for Apache Fluo. It contains all of the
+ default configuration and scripts required to run Fluo applications in YARN.</description>
+ <!-- NOTE: These dependencies are bundled in this assembly -->
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.fluo</groupId>
+ <artifactId>fluo-yarn-core</artifactId>
+ <optional>true</optional>
+ </dependency>
+ </dependencies>
+ <build>
+ <plugins>
+ <plugin>
+ <artifactId>maven-assembly-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>bin-assembly</id>
+ <goals>
+ <goal>single</goal>
+ </goals>
+ <phase>package</phase>
+ <configuration>
+ <escapeString>\</escapeString>
+ <descriptors>
+ <descriptor>src/main/assembly/bin.xml</descriptor>
+ </descriptors>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+ <profiles>
+ <profile>
+ <!-- attach source release when it is created by the apache-release profile -->
+ <id>apache-release</id>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>build-helper-maven-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>attach-source-release-assembly</id>
+ <goals>
+ <goal>attach-artifact</goal>
+ </goals>
+ <configuration>
+ <artifacts>
+ <artifact>
+ <file>${project.parent.build.directory}/${project.artifactId}-${project.version}-source-release.tar.gz</file>
+ <type>tar.gz</type>
+ <classifier>source-release</classifier>
+ </artifact>
+ </artifacts>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ </profiles>
+</project>
diff --git a/distribution/src/main/assembly/bin.xml b/distribution/src/main/assembly/bin.xml
new file mode 100644
index 0000000..68e27b8
--- /dev/null
+++ b/distribution/src/main/assembly/bin.xml
@@ -0,0 +1,73 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ agreements. See the NOTICE file distributed with this work for additional information regarding
+ copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance with the License. You may obtain a
+ copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software distributed under the License
+ is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ or implied. See the License for the specific language governing permissions and limitations under
+ the License.
+-->
+<assembly>
+ <id>bin</id>
+ <formats>
+ <format>tar.gz</format>
+ </formats>
+
+ <dependencySets>
+ <dependencySet>
+ <outputDirectory>lib</outputDirectory>
+ <directoryMode>0755</directoryMode>
+ <fileMode>0644</fileMode>
+ <useProjectArtifact>false</useProjectArtifact>
+ <useTransitiveDependencies>false</useTransitiveDependencies>
+ <includes>
+ <include>${groupId}:fluo-*</include>
+ </includes>
+ </dependencySet>
+ </dependencySets>
+
+ <fileSets>
+ <fileSet>
+ <directory>../</directory>
+ <outputDirectory>.</outputDirectory>
+ <includes>
+ <include>DISCLAIMER</include>
+ <include>LICENSE</include>
+ <include>NOTICE</include>
+ <include>README.md</include>
+ </includes>
+ </fileSet>
+ <fileSet>
+ <directory>bin</directory>
+ <outputDirectory>bin</outputDirectory>
+ <fileMode>755</fileMode>
+ <excludes>
+ <exclude>bin/fluo-yarn</exclude>
+ </excludes>
+ </fileSet>
+ <fileSet>
+ <directory>conf</directory>
+ <outputDirectory>conf</outputDirectory>
+ </fileSet>
+ <fileSet>
+ <directory>lib</directory>
+ <outputDirectory>lib</outputDirectory>
+ <fileMode>755</fileMode>
+ <includes>
+ <include>fetch.sh</include>
+ </includes>
+ </fileSet>
+ </fileSets>
+ <files>
+ <file>
+ <source>bin/fluo-yarn</source>
+ <outputDirectory>bin</outputDirectory>
+ <filtered>true</filtered>
+ </file>
+ </files>
+</assembly>
diff --git a/pom.xml b/pom.xml
new file mode 100644
index 0000000..952957a
--- /dev/null
+++ b/pom.xml
@@ -0,0 +1,241 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ agreements. See the NOTICE file distributed with this work for additional information regarding
+ copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance with the License. You may obtain a
+ copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software distributed under the License
+ is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ or implied. See the License for the specific language governing permissions and limitations under
+ the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.fluo</groupId>
+ <artifactId>fluo-parent</artifactId>
+ <version>1-incubating</version>
+ </parent>
+ <artifactId>fluo-yarn-project</artifactId>
+ <version>1.0.0-SNAPSHOT</version>
+ <packaging>pom</packaging>
+ <name>Apache Fluo YARN launcher</name>
+ <description>Tool that launches Apache Fluo application in Hadoop YARN</description>
+ <licenses>
+ <license>
+ <name>Apache License, Version 2.0</name>
+ <url>http://www.apache.org/licenses/LICENSE-2.0</url>
+ <distribution>repo</distribution>
+ </license>
+ </licenses>
+ <modules>
+ <module>distribution</module>
+ <module>core</module>
+ </modules>
+ <properties>
+ <accumulo.version>1.6.5</accumulo.version>
+ <findbugs.maxRank>9</findbugs.maxRank>
+ <fluo.version>1.2.0-SNAPSHOT</fluo.version>
+ <hadoop.version>2.6.3</hadoop.version>
+ <logback.version>1.1.3</logback.version>
+ <slf4j.version>1.7.12</slf4j.version>
+ <twill.version>0.11.0</twill.version>
+ </properties>
+ <dependencyManagement>
+ <dependencies>
+ <dependency>
+ <groupId>ch.qos.logback</groupId>
+ <artifactId>logback-classic</artifactId>
+ <version>${logback.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>ch.qos.logback</groupId>
+ <artifactId>logback-core</artifactId>
+ <version>${logback.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>com.google.code.gson</groupId>
+ <artifactId>gson</artifactId>
+ <version>2.8.0</version>
+ </dependency>
+ <dependency>
+ <!-- Guava 13.0.1 is required by Twill (due to beta method usage) -->
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ <version>13.0.1</version>
+ </dependency>
+ <dependency>
+ <!-- Required for Twill. See https://github.com/apache/fluo/issues/820 -->
+ <groupId>io.netty</groupId>
+ <artifactId>netty</artifactId>
+ <version>3.9.9.Final</version>
+ </dependency>
+ <dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ <version>4.12</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.fluo</groupId>
+ <artifactId>fluo-api</artifactId>
+ <version>${fluo.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.fluo</groupId>
+ <artifactId>fluo-command</artifactId>
+ <version>${fluo.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.fluo</groupId>
+ <artifactId>fluo-core</artifactId>
+ <version>${fluo.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.fluo</groupId>
+ <artifactId>fluo-yarn-core</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ <version>${hadoop.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.twill</groupId>
+ <artifactId>twill-api</artifactId>
+ <version>${twill.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.twill</groupId>
+ <artifactId>twill-ext</artifactId>
+ <version>${twill.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.twill</groupId>
+ <artifactId>twill-yarn</artifactId>
+ <version>${twill.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-api</artifactId>
+ <version>${slf4j.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-log4j12</artifactId>
+ <version>${slf4j.version}</version>
+ </dependency>
+ </dependencies>
+ </dependencyManagement>
+ <build>
+ <pluginManagement>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.accumulo</groupId>
+ <artifactId>accumulo-maven-plugin</artifactId>
+ <version>${accumulo.version}</version>
+ <configuration>
+ <instanceName>it-instance-maven</instanceName>
+ <rootPassword>ITSecret</rootPassword>
+ </configuration>
+ <executions>
+ <execution>
+ <id>run-plugin</id>
+ <goals>
+ <goal>start</goal>
+ <goal>stop</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.rat</groupId>
+ <artifactId>apache-rat-plugin</artifactId>
+ <configuration />
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-failsafe-plugin</artifactId>
+ <configuration>
+ <systemPropertyVariables>
+ <fluo.it.instance.name>it-instance-maven</fluo.it.instance.name>
+ <fluo.it.instance.clear>false</fluo.it.instance.clear>
+ </systemPropertyVariables>
+ </configuration>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-release-plugin</artifactId>
+ <configuration>
+ <!-- must put this here, due to MRELEASE-963 -->
+ <tagNameFormat>rel/fluo-@{project.version}</tagNameFormat>
+ </configuration>
+ </plugin>
+ </plugins>
+ </pluginManagement>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-dependency-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>analyze</id>
+ <goals>
+ <goal>analyze-only</goal>
+ </goals>
+ <configuration>
+ <failOnWarning>true</failOnWarning>
+ <ignoredDependencies>
+ <ignoredDependency>log4j:log4j:jar:*</ignoredDependency>
+ <ignoredDependency>org.apache.fluo:fluo-core:jar:${fluo.version}</ignoredDependency>
+ <ignoredDependency>org.apache.hadoop:hadoop-common:jar:${hadoop.version}</ignoredDependency>
+ <ignoredDependency>org.apache.hadoop:hadoop-client:jar:${hadoop.version}</ignoredDependency>
+ <ignoredDependency>org.apache.hadoop:hadoop-mapreduce-client-core:jar:${hadoop.version}</ignoredDependency>
+ <ignoredDependency>org.apache.hadoop:hadoop-yarn-api:jar:${hadoop.version}</ignoredDependency>
+ <ignoredDependency>org.apache.hadoop:hadoop-yarn-client:jar:${hadoop.version}</ignoredDependency>
+ <ignoredDependency>org.slf4j:slf4j-log4j12:jar:${slf4j.version}</ignoredDependency>
+ </ignoredDependencies>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+ <profiles>
+ <profile>
+ <id>fluo-release</id>
+ <!-- some properties to make the release build a bit faster -->
+ <properties>
+ <checkstyle.skip>true</checkstyle.skip>
+ <findbugs.skip>true</findbugs.skip>
+ <modernizer.skip>true</modernizer.skip>
+ <skipITs>true</skipITs>
+ <skipTests>true</skipTests>
+ </properties>
+ </profile>
+ <profile>
+ <!-- set proper source assembly name with apache-release and don't attach here -->
+ <id>apache-release</id>
+ <build>
+ <pluginManagement>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-assembly-plugin</artifactId>
+ <inherited>false</inherited>
+ <configuration>
+ <!-- source assembly gets attached in the assemble module -->
+ <attach>false</attach>
+ <finalName>fluo-${project.version}</finalName>
+ </configuration>
+ </plugin>
+ </plugins>
+ </pluginManagement>
+ </build>
+ </profile>
+ </profiles>
+</project>
