| <!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>The ECO API for Java · Apache Heron</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="<!--"/><meta name="docsearch:version" content="0.20.5-incubating"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="The ECO API for Java · Apache Heron"/><meta property="og:type" content="website"/><meta property="og:url" content="https://heron.apache.org/"/><meta property="og:description" content="<!--"/><meta property="og:image" content="https://heron.apache.org/img/undraw_online.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://heron.apache.org/img/undraw_tweetstorm.svg"/><link rel="shortcut icon" href="/img/favicon-32x32.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><link rel="alternate" type="application/atom+xml" href="https://heron.apache.org/blog/atom.xml" title="Apache Heron Blog ATOM Feed"/><link rel="alternate" type="application/rss+xml" href="https://heron.apache.org/blog/feed.xml" title="Apache Heron Blog RSS Feed"/><script> |
| (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ |
| (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), |
| m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) |
| })(window,document,'script','https://www.google-analytics.com/analytics.js','ga'); |
| |
| ga('create', 'UA-198017384-1', 'auto'); |
| ga('send', 'pageview'); |
| </script><script type="text/javascript" src="https://buttons.github.io/buttons.js"></script><script type="text/javascript" src="/js/custom.js"></script><script type="text/javascript" src="/js/fix-location.js"></script><link rel="stylesheet" href="/css/main.css"/><script src="/js/codetabs.js"></script></head><body class="sideNavVisible separateOnPageNav"><div class="fixedHeaderContainer"><div class="headerWrapper wrapper"><header><a href="/"><img class="logo" src="/img/HeronTextLogo-small.png" alt="Apache Heron"/><h2 class="headerTitleWithLogo">Apache Heron</h2></a><a href="/versions"><h3>0.20.5-incubating</h3></a><div class="navigationWrapper navigationSlider"><nav class="slidingNav"><ul class="nav-site nav-site-internal"><li class=""><a href="/api/java" target="_self">Javadocs</a></li><li class=""><a href="/api/python" target="_self">Pydocs</a></li><li class="siteNavGroupActive"><a href="/docs/getting-started-local-single-node" target="_self">Docs</a></li><li class=""><a href="/download" target="_self">Downloads</a></li><li class=""><a href="#community" target="_self">Community</a></li><li class=""><a href="/blog/" target="_self">Blog</a></li><li class=""><a href="#apache" target="_self">Apache</a></li></ul></nav></div></header></div></div><div class="navPusher"><div class="docMainWrapper wrapper"><div class="container docsNavContainer" id="docsNav"><nav class="toc"><div class="toggleNav"><section class="navWrapper wrapper"><div class="navBreadcrumb wrapper"><div class="navToggle" id="navToggler"><div class="hamburger-menu"><div class="line1"></div><div class="line2"></div><div class="line3"></div></div></div><h2><i>›</i><span>Topology Development APIs</span></h2><div class="tocToggler" id="tocToggler"><i class="icon-toc"></i></div></div><div class="navGroups"><div class="navGroup"><h3 class="navGroupCategoryTitle">Getting Started</h3><ul class=""><li class="navListItem"><a class="navItem" href="/docs/getting-started-local-single-node">Local (Single Node)</a></li><li class="navListItem"><a class="navItem" href="/docs/getting-started-migrate-storm-topologies">Migrate Storm Topologies</a></li><li class="navListItem"><a class="navItem" href="/docs/getting-started-docker">Heron & Docker</a></li><li class="navListItem"><a class="navItem" href="/docs/getting-started-troubleshooting-guide">Troubleshooting Guide</a></li></ul></div><div class="navGroup"><h3 class="navGroupCategoryTitle">Deployment</h3><ul class=""><li class="navListItem"><a class="navItem" href="/docs/deployment-overview">Deployment Overiew</a></li><li class="navListItem"><a class="navItem" href="/docs/deployment-configuration">Configuration</a></li><li class="navListItem"><a class="navItem" href="/docs/deployment-api-server">The Heron API Server</a></li></ul></div><div class="navGroup"><h3 class="navGroupCategoryTitle">Topology Development APIs</h3><ul class=""><li class="navListItem"><a class="navItem" href="/docs/topology-development-streamlet-api">The Heron Streamlet API for Java</a></li><li class="navListItem navListItemActive"><a class="navItem" href="/docs/topology-development-eco-api">The ECO API for Java</a></li><li class="navListItem"><a class="navItem" href="/docs/topology-development-topology-api-java">The Heron Topology API for Java</a></li><li class="navListItem"><a class="navItem" href="/docs/topology-development-topology-api-python">The Heron Topology API for Python</a></li><li class="navListItem"><a class="navItem" href="/docs/topology-development-streamlet-scala">The Heron Streamlet API for Scala</a></li></ul></div><div class="navGroup"><h3 class="navGroupCategoryTitle">Client API Docs</h3><ul class=""><li class="navListItem"><a class="navItem" href="/docs/client-api-docs-overview">Client API Docs</a></li></ul></div><div class="navGroup"><h3 class="navGroupCategoryTitle">Guides</h3><ul class=""><li class="navListItem"><a class="navItem" href="/docs/guides-effectively-once-java-topologies">Effectively Once Java Topologies</a></li><li class="navListItem"><a class="navItem" href="/docs/guides-data-model">Heron Data Model</a></li><li class="navListItem"><a class="navItem" href="/docs/guides-tuple-serialization">Tuple Serialization</a></li><li class="navListItem"><a class="navItem" href="/docs/guides-ui-guide">Heron UI Guide</a></li><li class="navListItem"><a class="navItem" href="/docs/guides-topology-tuning">Topology Tuning Guide</a></li><li class="navListItem"><a class="navItem" href="/docs/guides-packing-algorithms">Packing Algorithms</a></li><li class="navListItem"><a class="navItem" href="/docs/guides-simulator-mode">Simulator Mode</a></li><li class="navListItem"><a class="navItem" href="/docs/guides-troubeshooting-guide">Topology Troubleshooting Guide</a></li></ul></div><div class="navGroup"><h3 class="navGroupCategoryTitle">Heron Concepts</h3><ul class=""><li class="navListItem"><a class="navItem" href="/docs/heron-design-goals">Heron Design Goals</a></li><li class="navListItem"><a class="navItem" href="/docs/heron-topology-concepts">Heron Topologies</a></li><li class="navListItem"><a class="navItem" href="/docs/heron-streamlet-concepts">Heron Streamlets</a></li><li class="navListItem"><a class="navItem" href="/docs/heron-architecture">Heron Architecture</a></li><li class="navListItem"><a class="navItem" href="/docs/heron-delivery-semantics">Heron Delivery Semantics</a></li></ul></div><div class="navGroup"><h3 class="navGroupCategoryTitle">State Managers</h3><ul class=""><li class="navListItem"><a class="navItem" href="/docs/state-managers-zookeeper">Zookeeper</a></li><li class="navListItem"><a class="navItem" href="/docs/state-managers-local-fs">Local File System</a></li></ul></div><div class="navGroup"><h3 class="navGroupCategoryTitle">Uploaders</h3><ul class=""><li class="navListItem"><a class="navItem" href="/docs/uploaders-local-fs">Local File System</a></li><li class="navListItem"><a class="navItem" href="/docs/uploaders-hdfs">HDFS</a></li><li class="navListItem"><a class="navItem" href="/docs/uploaders-http">HTTP</a></li><li class="navListItem"><a class="navItem" href="/docs/uploaders-amazon-s3">Amazon S3</a></li><li class="navListItem"><a class="navItem" href="/docs/uploaders-scp">Secure Copy (SCP)</a></li></ul></div><div class="navGroup"><h3 class="navGroupCategoryTitle">Schedulers</h3><ul class=""><li class="navListItem"><a class="navItem" href="/docs/schedulers-k8s-by-hand">Kubernetes by hand</a></li><li class="navListItem"><a class="navItem" href="/docs/schedulers-k8s-with-helm">Kubernetes with Helm</a></li><li class="navListItem"><a class="navItem" href="/docs/schedulers-k8s-execution-environment">Kubernetes Environment Customization</a></li><li class="navListItem"><a class="navItem" href="/docs/schedulers-aurora-cluster">Aurora Cluster</a></li><li class="navListItem"><a class="navItem" href="/docs/schedulers-aurora-local">Aurora Locally</a></li><li class="navListItem"><a class="navItem" href="/docs/schedulers-local">Local Cluster</a></li><li class="navListItem"><a class="navItem" href="/docs/schedulers-nomad">Nomad</a></li><li class="navListItem"><a class="navItem" href="/docs/schedulers-mesos-local-mac">Mesos Cluster Locally</a></li><li class="navListItem"><a class="navItem" href="/docs/schedulers-slurm">Slurm Cluster</a></li><li class="navListItem"><a class="navItem" href="/docs/schedulers-yarn">YARN Cluster</a></li></ul></div><div class="navGroup"><h3 class="navGroupCategoryTitle">Cluster Configuration</h3><ul class=""><li class="navListItem"><a class="navItem" href="/docs/cluster-config-overview">Cluster Config Overview</a></li><li class="navListItem"><a class="navItem" href="/docs/cluster-config-system-level">System Level Configuration</a></li><li class="navListItem"><a class="navItem" href="/docs/cluster-config-instance">Heron Instance</a></li><li class="navListItem"><a class="navItem" href="/docs/cluster-config-metrics">Metrics Manager</a></li><li class="navListItem"><a class="navItem" href="/docs/cluster-config-stream">Stream Manager</a></li><li class="navListItem"><a class="navItem" href="/docs/cluster-config-tmanager">Topology Manager</a></li></ul></div><div class="navGroup"><h3 class="navGroupCategoryTitle">Observability</h3><ul class=""><li class="navListItem"><a class="navItem" href="/docs/observability-prometheus">Prometheus</a></li><li class="navListItem"><a class="navItem" href="/docs/observability-graphite">Graphite</a></li><li class="navListItem"><a class="navItem" href="/docs/observability-scribe">Scribe</a></li></ul></div><div class="navGroup"><h3 class="navGroupCategoryTitle">User Manuals</h3><ul class=""><li class="navListItem"><a class="navItem" href="/docs/user-manuals-heron-cli">Heron Client</a></li><li class="navListItem"><a class="navItem" href="/docs/user-manuals-heron-explorer">Heron Explorer</a></li><li class="navListItem"><a class="navItem" href="/docs/user-manuals-tracker-rest">Heron Tracker REST API</a></li><li class="navListItem"><a class="navItem" href="/docs/user-manuals-heron-tracker-runbook">Heron Tracker Runbook</a></li><li class="navListItem"><a class="navItem" href="/docs/user-manuals-heron-ui-runbook">Heron UI Runbook</a></li><li class="navListItem"><a class="navItem" href="/docs/user-manuals-heron-shell">Heron Shell</a></li></ul></div><div class="navGroup"><h3 class="navGroupCategoryTitle">Compiling</h3><ul class=""><li class="navListItem"><a class="navItem" href="/docs/compiling-overview">Compiling Overview</a></li><li class="navListItem"><a class="navItem" href="/docs/compiling-linux">Compiling on Linux</a></li><li class="navListItem"><a class="navItem" href="/docs/compiling-osx">Compiling on OS X</a></li><li class="navListItem"><a class="navItem" href="/docs/compiling-docker">Compiling With Docker</a></li><li class="navListItem"><a class="navItem" href="/docs/compiling-running-tests">Running Tests</a></li><li class="navListItem"><a class="navItem" href="/docs/compiling-code-organization">Code Organization</a></li></ul></div><div class="navGroup"><h3 class="navGroupCategoryTitle">Extending Heron</h3><ul class=""><li class="navListItem"><a class="navItem" href="/docs/extending-heron-scheduler">Custom Scheduler</a></li><li class="navListItem"><a class="navItem" href="/docs/extending-heron-metric-sink">Custom Metrics Sink</a></li></ul></div><div class="navGroup"><h3 class="navGroupCategoryTitle">Heron Resources</h3><ul class=""><li class="navListItem"><a class="navItem" href="/docs/heron-resources-resources">Heron Resources</a></li></ul></div></div></section></div><script> |
| var coll = document.getElementsByClassName('collapsible'); |
| var checkActiveCategory = true; |
| for (var i = 0; i < coll.length; i++) { |
| var links = coll[i].nextElementSibling.getElementsByTagName('*'); |
| if (checkActiveCategory){ |
| for (var j = 0; j < links.length; j++) { |
| if (links[j].classList.contains('navListItemActive')){ |
| coll[i].nextElementSibling.classList.toggle('hide'); |
| coll[i].childNodes[1].classList.toggle('rotate'); |
| checkActiveCategory = false; |
| break; |
| } |
| } |
| } |
| |
| coll[i].addEventListener('click', function() { |
| var arrow = this.childNodes[1]; |
| arrow.classList.toggle('rotate'); |
| var content = this.nextElementSibling; |
| content.classList.toggle('hide'); |
| }); |
| } |
| |
| document.addEventListener('DOMContentLoaded', function() { |
| createToggler('#navToggler', '#docsNav', 'docsSliderActive'); |
| createToggler('#tocToggler', 'body', 'tocActive'); |
| |
| var headings = document.querySelector('.toc-headings'); |
| headings && headings.addEventListener('click', function(event) { |
| var el = event.target; |
| while(el !== headings){ |
| if (el.tagName === 'A') { |
| document.body.classList.remove('tocActive'); |
| break; |
| } else{ |
| el = el.parentNode; |
| } |
| } |
| }, false); |
| |
| function createToggler(togglerSelector, targetSelector, className) { |
| var toggler = document.querySelector(togglerSelector); |
| var target = document.querySelector(targetSelector); |
| |
| if (!toggler) { |
| return; |
| } |
| |
| toggler.onclick = function(event) { |
| event.preventDefault(); |
| |
| target.classList.toggle(className); |
| }; |
| } |
| }); |
| </script></nav></div><div class="container mainContainer"><div class="wrapper"><div class="post"><header class="postHeader"><h1 class="postHeaderTitle">The ECO API for Java</h1></header><article><div><span><!-- |
| Licensed to the Apache Software Foundation (ASF) under one |
| or more contributor license agreements. See the NOTICE file |
| distributed with this work for additional information |
| regarding copyright ownership. The ASF licenses this file |
| to you under the Apache License, Version 2.0 (the |
| "License"); you may not use this file except in compliance |
| with the License. You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, |
| software distributed under the License is distributed on an |
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| KIND, either express or implied. See the License for the |
| specific language governing permissions and limitations |
| under the License. |
| --> |
| <blockquote> |
| <p><strong>The Heron ECO API is in beta</strong>. The Heron ECO API can be used to build and test topologies on your local or on a cluster. The API still needs some testing and feedback from the community to understand how we should continue to develop ECO.</p> |
| </blockquote> |
| <p>Heron processing topologies can be written using an API called the <strong>Heron ECO API</strong>. The ECO API is currently available to work with spouts and bolts from the following packages:</p> |
| <ul> |
| <li><code>org.apache.storm</code></li> |
| <li><code>org.apache.heron</code></li> |
| </ul> |
| <blockquote> |
| <p>Although this document focuses on the ECO API, both the <a href="heron-streamlet-concepts">Streamlet API</a> and <a href="heron-topology-concepts">Topology API</a> topologies you have built can still be used with Heron</p> |
| </blockquote> |
| <h2><a class="anchor" aria-hidden="true" id="the-heron-eco-api-vs-the-streamlet-and-topology-apis"></a><a href="#the-heron-eco-api-vs-the-streamlet-and-topology-apis" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>The Heron ECO API vs. The Streamlet and Topology APIs</h2> |
| <p>Heron's ECO offers one major difference over the Streamlet and Topology APIs and that is extensibility without recompilation. |
| With Heron's ECO developers now have a way to alter the way data flows through spouts and bolts without needing to get into their code and make changes. |
| Topologies can now be defined through a YAML based format.</p> |
| <h2><a class="anchor" aria-hidden="true" id="why-the-name-eco"></a><a href="#why-the-name-eco" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Why the name ECO?</h2> |
| <p>/ˈekoʊ/ (Because all software should come with a pronunciation guide these days) |
| ECO is an acronym that stands for:</p> |
| <ul> |
| <li>Extensible</li> |
| <li>Component</li> |
| <li>Orchestrator</li> |
| </ul> |
| <h2><a class="anchor" aria-hidden="true" id="what-about-storm-flux-is-it-compatible-with-eco"></a><a href="#what-about-storm-flux-is-it-compatible-with-eco" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>What about Storm Flux? Is it compatible with Eco?</h2> |
| <p>ECO is an extension of Flux. Most Storm Flux topologies should be able to deployed in Heron with minimal changes. |
| Start reading [Migrate Storm Topologies To Heron] (../../../migrate-storm-to-heron) to learn how to migrate your Storm Flux topology then come back.</p> |
| <h2><a class="anchor" aria-hidden="true" id="getting-started"></a><a href="#getting-started" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Getting started</h2> |
| <p>In order to use the Heron ECO API for Java, you'll need to install the <code>heron-api</code> and the <code>heron-storm</code> library, which is available |
| via <a href="http://search.maven.org/">Maven Central</a>.</p> |
| <h3><a class="anchor" aria-hidden="true" id="maven-setup"></a><a href="#maven-setup" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Maven setup</h3> |
| <p>To install the <code>heron-api</code> library using Maven, add this to the <code>dependencies</code> block of your <code>pom.xml</code> |
| configuration file:</p> |
| <pre><code class="hljs css language-xml"><span class="hljs-tag"><<span class="hljs-name">dependency</span>></span> |
| <span class="hljs-tag"><<span class="hljs-name">groupId</span>></span>org.apache.heron<span class="hljs-tag"></<span class="hljs-name">groupId</span>></span> |
| <span class="hljs-tag"><<span class="hljs-name">artifactId</span>></span>heron-api<span class="hljs-tag"></<span class="hljs-name">artifactId</span>></span> |
| <span class="hljs-tag"><<span class="hljs-name">version</span>></span>{{<span class="hljs-tag">< <span class="hljs-attr">heronVersion</span> ></span>}}<span class="hljs-tag"></<span class="hljs-name">version</span>></span> |
| <span class="hljs-tag"><<span class="hljs-name">scope</span>></span>compile<span class="hljs-tag"></<span class="hljs-name">scope</span>></span> |
| <span class="hljs-tag"></<span class="hljs-name">dependency</span>></span> |
| <span class="hljs-tag"><<span class="hljs-name">dependency</span>></span> |
| <span class="hljs-tag"><<span class="hljs-name">groupId</span>></span>org.apache.heron<span class="hljs-tag"></<span class="hljs-name">groupId</span>></span> |
| <span class="hljs-tag"><<span class="hljs-name">artifactId</span>></span>heron-storm<span class="hljs-tag"></<span class="hljs-name">artifactId</span>></span> |
| <span class="hljs-tag"><<span class="hljs-name">version</span>></span>{{<span class="hljs-tag">< <span class="hljs-attr">heronVersion</span> ></span>}}<span class="hljs-tag"></<span class="hljs-name">version</span>></span> |
| <span class="hljs-tag"><<span class="hljs-name">scope</span>></span>compile<span class="hljs-tag"></<span class="hljs-name">scope</span>></span> |
| <span class="hljs-tag"></<span class="hljs-name">dependency</span>></span> |
| </code></pre> |
| <h4><a class="anchor" aria-hidden="true" id="compiling-a-jar-with-dependencies"></a><a href="#compiling-a-jar-with-dependencies" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Compiling a JAR with dependencies</h4> |
| <p>In order to run a Java topology in a Heron cluster, you'll need to package your topology as a "fat" JAR with dependencies included. You can use the <a href="https://maven.apache.org/plugins/maven-assembly-plugin/usage.html">Maven Assembly Plugin</a> to generate JARs with dependencies. To install the plugin and add a Maven goal for a single JAR, add this to the <code>plugins</code> block in your <code>pom.xml</code>:</p> |
| <pre><code class="hljs css language-xml"><span class="hljs-tag"><<span class="hljs-name">plugin</span>></span> |
| <span class="hljs-tag"><<span class="hljs-name">artifactId</span>></span>maven-assembly-plugin<span class="hljs-tag"></<span class="hljs-name">artifactId</span>></span> |
| <span class="hljs-tag"><<span class="hljs-name">configuration</span>></span> |
| <span class="hljs-tag"><<span class="hljs-name">descriptorRefs</span>></span> |
| <span class="hljs-tag"><<span class="hljs-name">descriptorRef</span>></span>jar-with-dependencies<span class="hljs-tag"></<span class="hljs-name">descriptorRef</span>></span> |
| <span class="hljs-tag"></<span class="hljs-name">descriptorRefs</span>></span> |
| <span class="hljs-tag"><<span class="hljs-name">archive</span>></span> |
| <span class="hljs-tag"><<span class="hljs-name">manifest</span>></span> |
| <span class="hljs-tag"><<span class="hljs-name">mainClass</span>></span><span class="hljs-tag"></<span class="hljs-name">mainClass</span>></span> |
| <span class="hljs-tag"></<span class="hljs-name">manifest</span>></span> |
| <span class="hljs-tag"></<span class="hljs-name">archive</span>></span> |
| <span class="hljs-tag"></<span class="hljs-name">configuration</span>></span> |
| <span class="hljs-tag"><<span class="hljs-name">executions</span>></span> |
| <span class="hljs-tag"><<span class="hljs-name">execution</span>></span> |
| <span class="hljs-tag"><<span class="hljs-name">id</span>></span>make-assembly<span class="hljs-tag"></<span class="hljs-name">id</span>></span> |
| <span class="hljs-tag"><<span class="hljs-name">phase</span>></span>package<span class="hljs-tag"></<span class="hljs-name">phase</span>></span> |
| <span class="hljs-tag"><<span class="hljs-name">goals</span>></span> |
| <span class="hljs-tag"><<span class="hljs-name">goal</span>></span>single<span class="hljs-tag"></<span class="hljs-name">goal</span>></span> |
| <span class="hljs-tag"></<span class="hljs-name">goals</span>></span> |
| <span class="hljs-tag"></<span class="hljs-name">execution</span>></span> |
| <span class="hljs-tag"></<span class="hljs-name">executions</span>></span> |
| <span class="hljs-tag"></<span class="hljs-name">plugin</span>></span> |
| </code></pre> |
| <p>Once your <code>pom.xml</code> is properly set up, you can compile the JAR with dependencies using this command:</p> |
| <pre><code class="hljs css language-bash">$ mvn assembly:assembly |
| </code></pre> |
| <p>By default, this will add a JAR in your project's <code>target</code> folder with the name <code>PROJECT-NAME-VERSION-jar-with-dependencies.jar</code>. Here's an example ECO topology submission command using a compiled JAR:</p> |
| <pre><code class="hljs css language-bash">$ heron submit <span class="hljs-built_in">local</span> \ |
| target/my-project-1.2.3-jar-with-dependencies.jar \ |
| org.apache.heron.eco.Eco \ |
| --eco-config-file path/to/your/topology-definition.yaml |
| </code></pre> |
| <h3><a class="anchor" aria-hidden="true" id="reference-links"></a><a href="#reference-links" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Reference Links</h3> |
| <p><a href="#topology-name">Topology Name</a></p> |
| <p><a href="#configuration">Configuration</a></p> |
| <p><a href="#components">Components</a></p> |
| <p><a href="#property-injection">Property Injection</a></p> |
| <p><a href="#the-topology-definition">The Topology Definition</a></p> |
| <p><a href="#streams-and-groupings">Streams and Groupings</a></p> |
| <p><a href="#handling-enums">Handling Enums</a></p> |
| <p><a href="#property-substitution">Property Substitution</a></p> |
| <p><a href="#environment-variable-substitution">Environment Variable Substitution</a></p> |
| <p><a href="#other-eco-examples">Other ECO Examples</a></p> |
| <p>Notice how the above example submission command is referencing the main class <code>org.apache.heron.eco.Eco</code>. This part of the command |
| needs to stay the same. Eco is the main class that will assemble your topology from the <code>--eco-config-file</code> you specify.</p> |
| <h2><a class="anchor" aria-hidden="true" id="defining-your-eco-topology-file"></a><a href="#defining-your-eco-topology-file" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Defining Your ECO Topology File</h2> |
| <p>An ECO topology definition consists of the following:</p> |
| <ul> |
| <li>A topology name</li> |
| <li>An optional list of topology "components" (named Java objects that will be made available for configuration in the topology)</li> |
| <li>A DSL topology definition that contains: |
| <ul> |
| <li>A list of spouts, each identified by a unique ID</li> |
| <li>A list of bolts, each identified by a unique ID</li> |
| <li>A list of "stream" objects representing a flow of tuples between spouts and bolts</li> |
| </ul></li> |
| </ul> |
| <p>An example of a simple YAML DSL definition is below:</p> |
| <pre><code class="hljs css language-yaml"> |
| <span class="hljs-attr">name:</span> <span class="hljs-string">"fibonacci-topology"</span> |
| |
| <span class="hljs-attr">config:</span> |
| <span class="hljs-attr">topology.workers:</span> <span class="hljs-number">1</span> |
| |
| <span class="hljs-attr">components:</span> |
| <span class="hljs-bullet">-</span> <span class="hljs-attr">id:</span> <span class="hljs-string">"property-holder"</span> |
| <span class="hljs-attr">className:</span> <span class="hljs-string">"org.apache.heron.examples.eco.TestPropertyHolder"</span> |
| <span class="hljs-attr">constructorArgs:</span> |
| <span class="hljs-bullet">-</span> <span class="hljs-string">"some argument"</span> |
| <span class="hljs-attr">properties:</span> |
| <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">"numberProperty"</span> |
| <span class="hljs-attr">value:</span> <span class="hljs-number">11</span> |
| <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">"publicProperty"</span> |
| <span class="hljs-attr">value:</span> <span class="hljs-string">"This is public property"</span> |
| |
| <span class="hljs-attr">spouts:</span> |
| <span class="hljs-bullet">-</span> <span class="hljs-attr">id:</span> <span class="hljs-string">"spout-1"</span> |
| <span class="hljs-attr">className:</span> <span class="hljs-string">"org.apache.heron.examples.eco.TestFibonacciSpout"</span> |
| <span class="hljs-attr">constructorArgs:</span> |
| <span class="hljs-bullet">-</span> <span class="hljs-attr">ref:</span> <span class="hljs-string">"property-holder"</span> |
| <span class="hljs-attr">parallelism:</span> <span class="hljs-number">1</span> |
| |
| <span class="hljs-attr">bolts:</span> |
| <span class="hljs-bullet">-</span> <span class="hljs-attr">id:</span> <span class="hljs-string">"even-and-odd-bolt"</span> |
| <span class="hljs-attr">className:</span> <span class="hljs-string">"org.apache.heron.examples.eco.EvenAndOddBolt"</span> |
| <span class="hljs-attr">parallelism:</span> <span class="hljs-number">1</span> |
| |
| <span class="hljs-bullet">-</span> <span class="hljs-attr">id:</span> <span class="hljs-string">"ibasic-print-bolt"</span> |
| <span class="hljs-attr">className:</span> <span class="hljs-string">"org.apache.heron.examples.eco.TestIBasicPrintBolt"</span> |
| <span class="hljs-attr">parallelism:</span> <span class="hljs-number">1</span> |
| <span class="hljs-attr">configMethods:</span> |
| <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">"sampleConfigurationMethod"</span> |
| <span class="hljs-attr">args:</span> |
| <span class="hljs-bullet">-</span> <span class="hljs-string">"${ecoPropertyOne}"</span> |
| <span class="hljs-bullet">-</span> <span class="hljs-string">MB</span> |
| |
| <span class="hljs-bullet">-</span> <span class="hljs-attr">id:</span> <span class="hljs-string">"sys-out-bolt"</span> |
| <span class="hljs-attr">className:</span> <span class="hljs-string">"org.apache.heron.examples.eco.TestPrintBolt"</span> |
| <span class="hljs-attr">parallelism:</span> <span class="hljs-number">1</span> |
| |
| <span class="hljs-attr">streams:</span> |
| <span class="hljs-bullet">-</span> <span class="hljs-attr">from:</span> <span class="hljs-string">"spout-1"</span> |
| <span class="hljs-attr">to:</span> <span class="hljs-string">"even-and-odd-bolt"</span> |
| <span class="hljs-attr">grouping:</span> |
| <span class="hljs-attr">type:</span> <span class="hljs-string">SHUFFLE</span> |
| |
| <span class="hljs-bullet">-</span> <span class="hljs-attr">from:</span> <span class="hljs-string">"even-and-odd-bolt"</span> |
| <span class="hljs-attr">to:</span> <span class="hljs-string">"ibasic-print-bolt"</span> |
| <span class="hljs-attr">grouping:</span> |
| <span class="hljs-attr">type:</span> <span class="hljs-string">SHUFFLE</span> |
| <span class="hljs-attr">streamId:</span> <span class="hljs-string">"odds"</span> |
| |
| <span class="hljs-bullet">-</span> <span class="hljs-attr">from:</span> <span class="hljs-string">"even-and-odd-bolt"</span> |
| <span class="hljs-attr">to:</span> <span class="hljs-string">"sys-out-bolt"</span> |
| <span class="hljs-attr">grouping:</span> |
| <span class="hljs-attr">type:</span> <span class="hljs-string">SHUFFLE</span> |
| <span class="hljs-attr">streamId:</span> <span class="hljs-string">"evens"</span> |
| |
| </code></pre> |
| <p>If you want to stop here and try to deploy the above topology you can execute:</p> |
| <pre><code class="hljs css language-bash">$ heron submit <span class="hljs-built_in">local</span> \ |
| ~/.heron/examples/heron-eco-examples.jar \ |
| org.apache.heron.eco.Eco \ |
| --eco-config-file ~/.heron/examples/storm_fibonacci.yaml |
| </code></pre> |
| <p>This ECO topology does not do anything spectacular, but it's a good starting point to go through some of ECO's concepts.</p> |
| <h2><a class="anchor" aria-hidden="true" id="taking-a-closer-look-at-the-yaml-definition-specs"></a><a href="#taking-a-closer-look-at-the-yaml-definition-specs" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Taking a closer look at the YAML definition specs</h2> |
| <h3><a class="anchor" aria-hidden="true" id="topology-name"></a><a href="#topology-name" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Topology Name</h3> |
| <p>Each ECO definition file will be required to have a <code>name</code> defined.</p> |
| <pre><code class="hljs css language-yaml"> |
| <span class="hljs-attr">name:</span> <span class="hljs-string">"simple-wordcount-topology"</span> |
| |
| </code></pre> |
| <h3><a class="anchor" aria-hidden="true" id="configuration"></a><a href="#configuration" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Configuration</h3> |
| <p><code>config</code> is the section where you will list your properties to be inserted into a <code>org.apache.heron.api.Config</code> class. This section is optional.</p> |
| <pre><code class="hljs css language-yaml"> |
| <span class="hljs-attr">config:</span> |
| <span class="hljs-attr">topology.workers:</span> <span class="hljs-number">1</span> |
| |
| </code></pre> |
| <h4><a class="anchor" aria-hidden="true" id="specifying-component-level-resources"></a><a href="#specifying-component-level-resources" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Specifying Component Level Resources</h4> |
| <p>You can specify component level JVM resources by referencing the <code>id</code> of the component and its <code>ram</code>. |
| You can choose between Bytes <code>B</code>, Megabytes <code>MB</code>, or Gigabytes <code>GB</code>. Examples would be <code>256MB</code> or <code>2GB</code>. |
| The unit of measurement must be appended at the end of the numerical value with no spaces. There is plan to support |
| component level <code>cpu</code> and <code>disk</code> configs in the future.</p> |
| <pre><code class="hljs css language-yaml"> <span class="hljs-attr">topology.component.resourcemap:</span> |
| |
| <span class="hljs-bullet">-</span> <span class="hljs-attr">id:</span> <span class="hljs-string">"spout-1"</span> |
| <span class="hljs-attr">ram:</span> <span class="hljs-string">256MB</span> <span class="hljs-comment"># The minimum value for a component's specified RAM is 256MB</span> |
| |
| |
| <span class="hljs-bullet">-</span> <span class="hljs-attr">id:</span> <span class="hljs-string">"bolt-1"</span> |
| <span class="hljs-attr">ram:</span> <span class="hljs-string">256MB</span> <span class="hljs-comment"># The minimum value for a component's specified RAM is 256MB</span> |
| |
| </code></pre> |
| <h4><a class="anchor" aria-hidden="true" id="specifying-jvm-options"></a><a href="#specifying-jvm-options" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Specifying JVM Options</h4> |
| <p>You can specify component level JVM resources by referencing the <code>id</code> of the component and a list |
| of the JVM options</p> |
| <pre><code class="hljs css language-yaml"><span class="hljs-attr">topology.component.jvmoptions:</span> |
| |
| <span class="hljs-bullet">-</span> <span class="hljs-attr">id:</span> <span class="hljs-string">"spout-1"</span> |
| <span class="hljs-attr">options:</span> <span class="hljs-string">["-XX:NewSize=300m",</span> <span class="hljs-string">"-Xms2g"</span><span class="hljs-string">]</span> |
| </code></pre> |
| <h4><a class="anchor" aria-hidden="true" id="other-supported-configuration-parameters"></a><a href="#other-supported-configuration-parameters" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Other Supported Configuration Parameters</h4> |
| <ul> |
| <li><code>"topology.worker.childopts"</code> : Topology-specific options for the worker child process. This is used in addition to WORKER_CHILDOPTS</li> |
| <li><code>"topology.tick.tuple.freq.ms"</code> : How often (in milliseconds) a tick tuple from the "__system" component and "__tick" stream should be sent to tasks. Meant to be used as a component-specific configuration.</li> |
| <li><code>"topology.enable.message.timeouts"</code> : True if Heron should timeout messages or not. Defaults to true. This is meant to be used in unit tests to prevent tuples from being accidentally timed out during the test.</li> |
| <li><code>"topology.debug"</code> : When set to true, Heron will log every message that's emitted.</li> |
| <li><code>"topology.stmgrs"</code> : The number of stmgr instances that should spin up to service this topology. All the executors will be evenly shared by these stmgrs.</li> |
| <li><code>"topology.message.timeout.secs"</code> : The maximum amount of time given to the topology to fully process a message |
| emitted by a spout. If the message is not acked within this time frame, Heron |
| will fail the message on the spout. Some spouts implementations will then replay |
| the message at a later time.</li> |
| <li><code>"topology.component.parallelism"</code> : The per component parallelism for a component in this topology.</li> |
| <li><code>"topology.max.spout.pending"</code> : This config applies to individual tasks, not to spouts or topologies as a whole. |
| A pending tuple is one that has been emitted from a spout but has not been acked or failed yet. |
| Note that this config parameter has no effect for unreliable spouts that don't tag |
| their tuples with a message id.</li> |
| <li><code>"topology.auto.task.hooks"</code> : A list of task hooks that are automatically added to every spout and bolt in the topology. An example |
| of when you'd do this is to add a hook that integrates with your internal |
| monitoring system. These hooks are instantiated using the zero-arg constructor.</li> |
| <li><code>"topology.serializer.classname"</code> : The serialization class that is used to serialize/deserialize tuples</li> |
| <li><code>"topology.reliability.mode"</code> : A Heron topology can be run in any one of the TopologyReliabilityMode |
| mode. The format of this flag is the string encoded values of the |
| underlying TopologyReliabilityMode value. Values are <code>ATMOST_ONCE</code>, <code>ATLEAST_ONCE</code>, and <code>EFFECTIVELY_ONCE</code>.</li> |
| <li><code>"topology.reliability.mode"</code> : A Heron topology can be run in any one of the TopologyReliabilityMode |
| mode. The format of this flag is the string encoded values of the |
| underlying TopologyReliabilityMode value.</li> |
| <li><code>"topology.container.cpu"</code> : Number of CPU cores per container to be reserved for this topology.</li> |
| <li><code>"topology.container.ram"</code> : Amount of RAM per container to be reserved for this topology. In bytes.</li> |
| <li><code>"topology.container.disk"</code> : Amount of disk per container to be reserved for this topology. In bytes.</li> |
| <li><code>"topology.container.max.cpu.hint"</code> : Hint for max number of CPU cores per container to be reserved for this topology.</li> |
| <li><code>"topology.container.max.ram.hint"</code> : Hint for max amount of RAM per container to be reserved for this topology. In bytes.</li> |
| <li><code>"topology.container.max.disk.hint"</code> : Hint for max amount of disk per container to be reserved for this topology. In bytes.</li> |
| <li><code>"topology.container.padding.percentage"</code> : Hint for max amount of disk per container to be reserved for this topology. In bytes.</li> |
| <li><code>"topology.container.ram.padding"</code> : Amount of RAM to pad each container. In bytes.</li> |
| <li><code>"topology.stateful.checkpoint.interval.seconds"</code> : What's the checkpoint interval for stateful topologies in seconds.</li> |
| <li><code>"topology.stateful.start.clean"</code> : Boolean flag that says that the stateful topology should start from clean state, i.e. ignore any checkpoint state.</li> |
| <li><code>"topology.name"</code> : Name of the topology. This config is automatically set by Heron when the topology is submitted.</li> |
| <li><code>"topology.team.name"</code> : Name of the team which owns this topology.</li> |
| <li><code>"topology.team.email"</code> : Email of the team which owns this topology.</li> |
| <li><code>"topology.cap.ticket"</code> : Cap ticket (if filed) for the topology. If the topology is in prod this has to be set or it cannot be deployed.</li> |
| <li><code>"topology.project.name"</code> : Project name of the topology, to help us with tagging which topologies are part of which project. For example, if topology A and Topology B are part of the same project, we will like to aggregate them as part of the same project. This is required by Cap team.</li> |
| <li><code>"topology.additional.classpath"</code> : Any user defined classpath that needs to be passed to instances should be set in to config through this key. The value will be of the format "cp1:cp2:cp3..."</li> |
| <li><code>"topology.update.deactivate.wait.secs"</code> : Amount of time to wait after deactivating a topology before updating it</li> |
| <li><code>"topology.update.reactivate.wait.secs"</code> : fter updating a topology, amount of time to wait for it to come back up before reactivating it</li> |
| <li><code>"topology.environment"</code> : Topology-specific environment properties to be added to an Heron instance. This is added to the existing environment (that of the Heron instance). This variable contains Map<String, String></li> |
| <li><code>"topology.timer.events"</code> : Timer events registered for a topology. This is a Map<String, Pair<Duration, Runnable>>. Where the key is the name and the value contains the frequency of the event and the task to run.</li> |
| <li><code>"topology.remote.debugging.enable"</code> : Enable Remote debugging for java heron instances</li> |
| <li><code>"topology.droptuples.upon.backpressure"</code> : Do we want to drop tuples instead of initiating Spout BackPressure</li> |
| <li><code>"topology.component.output.bps"</code> : The per component output bytes per second in this topology</li> |
| </ul> |
| <h3><a class="anchor" aria-hidden="true" id="components"></a><a href="#components" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Components</h3> |
| <p><code>components</code> are a list of instances that would be used as configuration objects for other components in your ECO file defined in the YAML DSL. |
| The properties that are required for each <code>component</code> instance are <code>id</code> and <code>className</code>. <code>id</code> can be any name you choose, <code>className</code> is the fully qualified className of the Java class. The <code>id</code> field is used to identify |
| the component for injection in the coming spouts and bolts defined in the topology. <code>constructorArgs</code> is only needed |
| if a component has constructor that requires arguments. We will get into constructor args the in the next section. <code>components</code> are optional.</p> |
| <pre><code class="hljs css language-yaml"> |
| <span class="hljs-attr">components:</span> |
| <span class="hljs-bullet">-</span> <span class="hljs-attr">id:</span> <span class="hljs-string">"property-holder"</span> |
| <span class="hljs-attr">className:</span> <span class="hljs-string">"org.apache.heron.examples.eco.TestPropertyHolder"</span> |
| <span class="hljs-attr">constructorArgs:</span> |
| <span class="hljs-bullet">-</span> <span class="hljs-string">"some argument"</span> |
| |
| </code></pre> |
| <h3><a class="anchor" aria-hidden="true" id="property-injection"></a><a href="#property-injection" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Property Injection</h3> |
| <h4><a class="anchor" aria-hidden="true" id="constructor-injection"></a><a href="#constructor-injection" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Constructor Injection</h4> |
| <p><code>constructorArgs</code> can specify any object type. Above you can see that the only constructor argument specified is a string |
| that contained "some argument". If declaring a number, you may omit the parenthesis.</p> |
| <pre><code class="hljs css language-yaml"> |
| <span class="hljs-attr">constructorArgs:</span> |
| <span class="hljs-bullet">-</span> <span class="hljs-string">"some argument"</span> |
| <span class="hljs-bullet">-</span> <span class="hljs-number">123.45</span> |
| |
| </code></pre> |
| <p>The is also a way to reference other components as arguments by using <code>ref</code>. In the example |
| below we are specifying an already defined component to be a constructor argument. Any instance that is referenced by |
| <code>ref</code> must have already been defined in the ECO definition file before it is to be used.</p> |
| <pre><code class="hljs css language-yaml"> |
| <span class="hljs-attr">constructorArgs:</span> |
| <span class="hljs-bullet">-</span> <span class="hljs-attr">ref:</span> <span class="hljs-string">"property-holder"</span> |
| |
| </code></pre> |
| <h4><a class="anchor" aria-hidden="true" id="setter-and-public-field-injection"></a><a href="#setter-and-public-field-injection" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Setter and Public Field Injection</h4> |
| <p>Besides constructor injection, you may also use setter methods. In the below example, ECO will take inspect the component |
| for setters that match the names and values provided. If no setter is defined, it will then look for a public field to set the property.</p> |
| <pre><code class="hljs css language-yaml"> |
| <span class="hljs-attr">properties:</span> |
| <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">"numberProperty"</span> |
| <span class="hljs-attr">value:</span> <span class="hljs-number">11</span> |
| <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">"publicProperty"</span> |
| <span class="hljs-attr">value:</span> <span class="hljs-string">"This is public property"</span> |
| |
| </code></pre> |
| <h2><a class="anchor" aria-hidden="true" id="the-topology-definition"></a><a href="#the-topology-definition" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>The Topology Definition</h2> |
| <p>Spouts and Bolts each have their own sections for defining in the ECO file. They are extensions of <code>components</code> so they will |
| be allowed the same property injection methods above. One difference is the <code>parallelism</code> property they contain, it sets the <code>parallelism</code> |
| property for each bolt or spout once the topology has been deployed into Heron.</p> |
| <h3><a class="anchor" aria-hidden="true" id="spouts"></a><a href="#spouts" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Spouts</h3> |
| <pre><code class="hljs css language-yaml"> |
| <span class="hljs-attr">spouts:</span> |
| <span class="hljs-bullet">-</span> <span class="hljs-attr">id:</span> <span class="hljs-string">"spout-1"</span> |
| <span class="hljs-attr">className:</span> <span class="hljs-string">"org.apache.heron.examples.eco.TestFibonacciSpout"</span> |
| <span class="hljs-attr">constructorArgs:</span> |
| <span class="hljs-bullet">-</span> <span class="hljs-attr">ref:</span> <span class="hljs-string">"property-holder"</span> |
| <span class="hljs-attr">parallelism:</span> <span class="hljs-number">1</span> |
| |
| </code></pre> |
| <h3><a class="anchor" aria-hidden="true" id="bolts"></a><a href="#bolts" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Bolts</h3> |
| <pre><code class="hljs css language-yaml"> |
| <span class="hljs-attr">bolts:</span> |
| <span class="hljs-bullet">-</span> <span class="hljs-attr">id:</span> <span class="hljs-string">"even-and-odd-bolt"</span> |
| <span class="hljs-attr">className:</span> <span class="hljs-string">"org.apache.heron.examples.eco.EvenAndOddBolt"</span> |
| <span class="hljs-attr">parallelism:</span> <span class="hljs-number">1</span> |
| |
| </code></pre> |
| <h3><a class="anchor" aria-hidden="true" id="streams-and-groupings"></a><a href="#streams-and-groupings" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Streams and Groupings</h3> |
| <p>Streams are what connect your bolts and spouts together. Stream Groupings are the specific way you are to connect those streams to the spouts and bolts.</p> |
| <h5><a class="anchor" aria-hidden="true" id="a-stream-can-contain-the-following-fields"></a><a href="#a-stream-can-contain-the-following-fields" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>A Stream can contain the following fields.</h5> |
| <ul> |
| <li><code>from</code> - references the <code>id</code> of the component where data is coming from</li> |
| <li><code>to</code> - references the <code>id</code> of the component where the data is going</li> |
| <li><code>grouping</code> - This is grouping definition of how this stream will connect the two afore mentioned components together</li> |
| </ul> |
| <h5><a class="anchor" aria-hidden="true" id="a-grouping-can-contain-the-following-fields"></a><a href="#a-grouping-can-contain-the-following-fields" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>A grouping can contain the following fields</h5> |
| <ul> |
| <li><code>type</code> - The type of grouping. |
| <ul> |
| <li><code>SHUFFLE</code></li> |
| <li><code>FIELDS</code></li> |
| <li><code>ALL</code></li> |
| <li><code>GLOBAL</code></li> |
| <li><code>NONE</code></li> |
| <li><code>CUSTOM</code></li> |
| </ul></li> |
| <li><code>args</code> is specific to the <code>FIELDS</code> grouping type. You would specify this as a list like so <code>["arg1", "arg2"]</code></li> |
| <li><code>customClass</code> if you wanted to create a custom grouping, you could specify the fully qualified class name here</li> |
| </ul> |
| <p>In the below example, you can see the first Stream Definition declares that data will flow from <code>spout-1</code> to <code>even-and-odd-bolt</code> with a grouping type of <code>SHUFFLE</code></p> |
| <pre><code class="hljs css language-yaml"> |
| <span class="hljs-attr">streams:</span> |
| <span class="hljs-bullet">-</span> <span class="hljs-attr">from:</span> <span class="hljs-string">"spout-1"</span> |
| <span class="hljs-attr">to:</span> <span class="hljs-string">"even-and-odd-bolt"</span> |
| <span class="hljs-attr">grouping:</span> |
| <span class="hljs-attr">type:</span> <span class="hljs-string">SHUFFLE</span> |
| |
| <span class="hljs-bullet">-</span> <span class="hljs-attr">from:</span> <span class="hljs-string">"even-and-odd-bolt"</span> |
| <span class="hljs-attr">to:</span> <span class="hljs-string">"ibasic-print-bolt"</span> |
| <span class="hljs-attr">grouping:</span> |
| <span class="hljs-attr">type:</span> <span class="hljs-string">SHUFFLE</span> |
| <span class="hljs-attr">streamId:</span> <span class="hljs-string">"odds"</span> |
| |
| <span class="hljs-bullet">-</span> <span class="hljs-attr">from:</span> <span class="hljs-string">"even-and-odd-bolt"</span> |
| <span class="hljs-attr">to:</span> <span class="hljs-string">"sys-out-bolt"</span> |
| <span class="hljs-attr">grouping:</span> |
| <span class="hljs-attr">type:</span> <span class="hljs-string">SHUFFLE</span> |
| <span class="hljs-attr">streamId:</span> <span class="hljs-string">"evens"</span> |
| |
| </code></pre> |
| <h3><a class="anchor" aria-hidden="true" id="handling-enums"></a><a href="#handling-enums" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Handling Enums</h3> |
| <p>The usage of Enums is supported in ECO. You can use enums in constructor arguments, references, |
| configuration methods, and properties In the <code>fibonacci-topology</code> referenced above. In the examples we reference the |
| enum <code>TestUnits</code>. The enum is shown below.</p> |
| <pre><code class="hljs css language-java"><span class="hljs-keyword">public</span> <span class="hljs-keyword">enum</span> TestUnits { |
| MB(<span class="hljs-string">"MB"</span>), |
| GB(<span class="hljs-string">"GB"</span>), |
| B(<span class="hljs-string">"B"</span>); |
| |
| String value; |
| |
| TestUnits(String value) { |
| <span class="hljs-keyword">this</span>.value = value; |
| } |
| |
| <span class="hljs-function"><span class="hljs-keyword">public</span> String <span class="hljs-title">getValue</span><span class="hljs-params">()</span> </span>{ |
| <span class="hljs-keyword">return</span> value; |
| } |
| } |
| </code></pre> |
| <p>In the <code>ibasic-print-bolt</code> a <code>configMethod</code> is specified with the name <code>sampleConfigurationMethod</code>.</p> |
| <pre><code class="hljs css language-yaml"> <span class="hljs-bullet">-</span> <span class="hljs-attr">id:</span> <span class="hljs-string">"ibasic-print-bolt"</span> |
| <span class="hljs-string">...</span> <span class="hljs-string">excluded</span> <span class="hljs-string">for</span> <span class="hljs-string">simplicity</span> |
| <span class="hljs-attr">args:</span> |
| <span class="hljs-bullet">-</span> <span class="hljs-string">"someStringArgument"</span> |
| <span class="hljs-bullet">-</span> <span class="hljs-string">MB</span> |
| </code></pre> |
| <p>This is the same as calling the following java method</p> |
| <pre><code class="hljs css language-java"> |
| <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">void</span> <span class="hljs-title">sampleConfigurationMethod</span><span class="hljs-params">(String someProperty, TestUnits TestUnits)</span> </span>{ |
| <span class="hljs-keyword">this</span>.someProperty += someProperty; |
| <span class="hljs-keyword">this</span>.TestUnits = TestUnits; |
| } |
| |
| </code></pre> |
| <h3><a class="anchor" aria-hidden="true" id="property-substitution"></a><a href="#property-substitution" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Property Substitution</h3> |
| <p>It's always nice to be able to define properties based on the environments you are in. We haven't forgotten this with ECO. |
| You are able to substitute values into your ECO files by either environment variables or a properties file.<br> |
| To turn on property substitution add the flag <code>--props</code> and specify a path to a <code>.properties</code> file.</p> |
| <p>To start over run</p> |
| <pre><code class="hljs css language-bash">$ heron <span class="hljs-built_in">kill</span> <span class="hljs-built_in">local</span> fibonacci-topology |
| </code></pre> |
| <p>After the topology has been killed, execute:</p> |
| <pre><code class="hljs css language-bash">$ heron submit <span class="hljs-built_in">local</span> \ |
| ~/.heron/examples/heron-eco-examples.jar \ |
| org.apache.heron.eco.Eco \ |
| --eco-config-file ~/.heron/examples/fibonacci.yaml --props ~/.heron/examples/sample.properties |
| </code></pre> |
| <p>If you look above at the yaml file snippet at the beginning of the page you will see</p> |
| <pre><code class="hljs css language-yaml"><span class="hljs-bullet">-</span> <span class="hljs-attr">id:</span> <span class="hljs-string">"ibasic-print-bolt"</span> |
| <span class="hljs-attr">className:</span> <span class="hljs-string">"org.apache.heron.examples.eco.TestIBasicPrintBolt"</span> |
| <span class="hljs-attr">parallelism:</span> <span class="hljs-number">1</span> |
| <span class="hljs-attr">configMethods:</span> |
| <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">"sampleConfigurationMethod"</span> |
| <span class="hljs-attr">args:</span> |
| <span class="hljs-bullet">-</span> <span class="hljs-string">"${ecoPropertyOne}"</span> |
| <span class="hljs-bullet">-</span> <span class="hljs-string">MB</span> |
| </code></pre> |
| <p>In the <code>sample.properties</code> we have a key value set at <code>ecoPropertyOne=thisValueWasSetFromAPropertiesFile</code>. You can check the logs |
| for the <code>ibasic-print-bolt</code> and see the values are printing out.</p> |
| <h3><a class="anchor" aria-hidden="true" id="environment-variable-substitution"></a><a href="#environment-variable-substitution" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Environment Variable Substitution</h3> |
| <p>ECO also allows you to do environment variable substitution. To activate environment variable substitution pass the flag <code>--env-props</code> upon submitting a topology. If you have <code>SOME_VARIABLE</code> defined |
| you can reference in your yaml file like below.</p> |
| <pre><code class="hljs css language-yaml"><span class="hljs-string">${ENV-SOME_VARIABLE}</span> |
| </code></pre> |
| <h3><a class="anchor" aria-hidden="true" id="other-eco-examples"></a><a href="#other-eco-examples" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Other ECO examples</h3> |
| <p>Run the simple wordcount example</p> |
| <pre><code class="hljs css language-bash">$ heron submit <span class="hljs-built_in">local</span> \ |
| ~/.heron/examples/heron-eco-examples.jar \ |
| org.apache.heron.eco.Eco \ |
| --eco-config-file ~/.heron/examples/storm_wordcount.yaml |
| </code></pre> |
| <p>Run the simple windowing example</p> |
| <pre><code class="hljs css language-bash">$ heron submit <span class="hljs-built_in">local</span> \ |
| ~/.heron/examples/heron-eco-examples.jar \ |
| org.apache.heron.eco.Eco \ |
| --eco-config-file ~/.heron/examples/storm_windowing.yaml |
| </code></pre> |
| </span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/topology-development-streamlet-api"><span class="arrow-prev">← </span><span>The Heron Streamlet API for Java</span></a><a class="docs-next button" href="/docs/topology-development-topology-api-java"><span>The Heron Topology API for Java</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#the-heron-eco-api-vs-the-streamlet-and-topology-apis">The Heron ECO API vs. The Streamlet and Topology APIs</a></li><li><a href="#why-the-name-eco">Why the name ECO?</a></li><li><a href="#what-about-storm-flux-is-it-compatible-with-eco">What about Storm Flux? Is it compatible with Eco?</a></li><li><a href="#getting-started">Getting started</a><ul class="toc-headings"><li><a href="#maven-setup">Maven setup</a></li><li><a href="#reference-links">Reference Links</a></li></ul></li><li><a href="#defining-your-eco-topology-file">Defining Your ECO Topology File</a></li><li><a href="#taking-a-closer-look-at-the-yaml-definition-specs">Taking a closer look at the YAML definition specs</a><ul class="toc-headings"><li><a href="#topology-name">Topology Name</a></li><li><a href="#configuration">Configuration</a></li><li><a href="#components">Components</a></li><li><a href="#property-injection">Property Injection</a></li></ul></li><li><a href="#the-topology-definition">The Topology Definition</a><ul class="toc-headings"><li><a href="#spouts">Spouts</a></li><li><a href="#bolts">Bolts</a></li><li><a href="#streams-and-groupings">Streams and Groupings</a></li><li><a href="#handling-enums">Handling Enums</a></li><li><a href="#property-substitution">Property Substitution</a></li><li><a href="#environment-variable-substitution">Environment Variable Substitution</a></li><li><a href="#other-eco-examples">Other ECO examples</a></li></ul></li></ul></nav></div><footer class="nav-footer" id="footer"><div class="apache-disclaimer">Apache Heron is an effort undergoing incubation at <a target="_blank" href="https://apache.org/">The Apache Software Foundation (ASF)</a> sponsored by the Apache Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.<br/><br/>Apache®, the names of Apache projects, and the feather logo are either <a rel="external" href="https://www.apache.org/foundation/marks/list/">registered trademarks or trademarks</a> of the Apache Software Foundation in the United States and/or other countries.<br/><br/><div class="copyright-box">Copyright © 2023 the Apache Software Foundation, Apache Heron, Heron, |
| Apache, the Apache feather Logo, and the Apache Heron project logo are either registered |
| trademarks or trademarks of the Apache Software Foundation.</div></div><div class="apache-links"><a class="item" rel="external" href="https://incubator.apache.org/">Apache Incubator</a><div><a class="item" rel="external" href="https://www.apache.org/">About the ASF</a></div><div><a class="item" rel="external" href="https://www.apache.org/events/current-event">Events</a></div><div><a class="item" rel="external" href="https://www.apache.org/foundation/thanks.html">Thanks</a></div><div><a class="item" rel="external" href="https://www.apache.org/foundation/sponsorship.html">Become a Sponsor</a></div><div><a class="item" rel="external" href="https://www.apache.org/security/">Security</a></div><div><a class="item" rel="external" href="https://www.apache.org/licenses/">License</a></div></div></footer></div><script>window.twttr=(function(d,s, id){var js,fjs=d.getElementsByTagName(s)[0],t=window.twttr||{};if(d.getElementById(id))return t;js=d.createElement(s);js.id=id;js.src='https://platform.twitter.com/widgets.js';fjs.parentNode.insertBefore(js, fjs);t._e = [];t.ready = function(f) {t._e.push(f);};return t;}(document, 'script', 'twitter-wjs'));</script></body></html> |
