This is a step by step guide to run Heron on a Mesos cluster locally.
Follow Quick Start Guide to install Heron.
Follow [Installing Mesos on your Mac with Homebrew] (https://mesosphere.com/blog/2014/07/07/installing-mesos-on-your-mac-with-homebrew/) to install and run Mesos. To confirm Mesos cluster is ready for accepting Heron topologies, access the Mesos management console http://localhost:5050 and confirm there is activated slaves.
By default, Heron uses Local File System State Manager on Mesos to manage states. Modify $HOME/.heron/conf/mesos/statemgr.yaml
to use ZooKeeper. For more details see Setting up ZooKeeper.
Heron needs to know where to load the lib to interact with Mesos. Change the config heron.mesos.native.library.path
in $HOME/.heron/conf/mesos/scheduler.yaml
to the library path of the Mesos install. If Mesos is installed through brew
, the library path should be /usr/local/Cellar/mesos/your_mesos_version/lib
.
Mesos only offers a C++ interface, which is not portable across platforms.
After setting up Heron and Mesos, submit a topology using the following command. By default this command loads the config in $HOME/.heron/conf
. Add --config-path=your_conf_path
to change the config path.
heron submit mesos --verbose ~/.heron/examples/heron-api-examples.jar \ org.apache.heron.examples.api.ExclamationTopology ExclamationTopology
The following will be displayed upon a successful submit.
[2016-07-25 22:04:41 -0700] org.apache.heron.scheduler.mesos.MesosLauncher INFO: \ For checking the status and logs of the topology, use the working directory \ $HOME/.herondata/topologies/mesos/$USER/ExclamationTopology [2016-07-25 22:04:41 -0700] org.apache.heron.scheduler.SubmitterMain FINE: Topology \ ExclamationTopology submitted successfully INFO: Topology 'ExclamationTopology' launched successfully INFO: Elapsed time: 4.114s.
Note that this doesn't necessarily mean the topology is successfully launched in Mesos, to verify check the working directory as shown in the output. You will see:
heron-examples.jar
: the jar which contains the topology submitted.heron-conf
: configurations used to launch the topology.log-files
: directory containing Mesos scheduler's log.The log file will show whether the launch succeeded. If it succeeded, at the end of the log file it will show the task is running.
[2016-07-25 22:15:47 -0700] org.apache.heron.scheduler.mesos.framework.MesosFramework INFO: \ Received status update [...] [2016-07-25 22:15:47 -0700] org.apache.heron.scheduler.mesos.framework.MesosFramework INFO: \ Task with id 'container_1_1469510147073:0' RUNNING
If the launch fails, an error message will be included. For example, if the Mesos library isn't found in the configured location, the following exception will occur.
[2016-07-25 22:04:42 -0700] stderr STDERR: Failed to load native Mesos library from \ /usr/lib/mesos/0.28.1/lib [2016-07-25 22:04:42 -0700] stderr STDERR: Exception in thread "main" [2016-07-25 22:04:42 -0700] stderr STDERR: java.lang.UnsatisfiedLinkError: no mesos in \ java.library.path [2016-07-25 22:04:42 -0700] stderr STDERR: at \ java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867) ...
Another way to check your topology is running is to look at the Mesos management console. If it was launched successfully, two containers will be running.
To view the process logs, click the sandbox
on the right side. The sandbox of the heron container is shown below.
The log-files
directory includes the application and GC log of the processes running in this container.
The bolt log of the ExclamationTopology is container_1_exclaim1_1.log.0
. Below is a sample of it.
Install Heron tools to monitor the topology with the heron-ui
(see [Quick Start Guide] (../../../../getting-started)). Configure the value of statemgrs.rootpath
in $HOME/.herontools/conf/heron_tracker.yaml
to $HOME/.herondata/repository/state/mesos
before starting the tracker. This configuration sets the location of the state manager root path. Start tracker and the UI.
$ heron-tracker ... Running on port: 8888 ... Using config file: $HOME/.herontools/conf/heron_tracker.yaml
$ heron-ui ... Running on port: 8889 ... Using tracker url: http://localhost:8888
Go to the UI at http://localhost:8889 to see the topology.
To see the metrics, click on the topology.
To enter the Mesos Management Console page, click the job
button.
To kill the topology, run:
heron kill mesos ExclamationTopology