| <!DOCTYPE html > |
| <html> |
| <head> |
| <title>source - org.apache.gearpump.streaming.source</title> |
| <meta name="description" content="source - org.apache.gearpump.streaming.source" /> |
| <meta name="keywords" content="source org.apache.gearpump.streaming.source" /> |
| <meta http-equiv="content-type" content="text/html; charset=UTF-8" /> |
| |
| <link href="../../../../../lib/template.css" media="screen" type="text/css" rel="stylesheet" /> |
| <link href="../../../../../lib/diagrams.css" media="screen" type="text/css" rel="stylesheet" id="diagrams-css" /> |
| <script type="text/javascript" src="../../../../../lib/jquery.js" id="jquery-js"></script> |
| <script type="text/javascript" src="../../../../../lib/jquery-ui.js"></script> |
| <script type="text/javascript" src="../../../../../lib/template.js"></script> |
| <script type="text/javascript" src="../../../../../lib/tools.tooltip.js"></script> |
| |
| <script type="text/javascript"> |
| if(top === self) { |
| var url = '../../../../../index.html'; |
| var hash = 'org.apache.gearpump.streaming.source.package'; |
| var anchor = window.location.hash; |
| var anchor_opt = ''; |
| if (anchor.length >= 1) |
| anchor_opt = '@' + anchor.substring(1); |
| window.location.href = url + '#' + hash + anchor_opt; |
| } |
| </script> |
| |
| </head> |
| <body class="value"> |
| <div id="definition"> |
| <img alt="Package" src="../../../../../lib/package_big.png" /> |
| <p id="owner"><a href="../../../../package.html" class="extype" name="org">org</a>.<a href="../../../package.html" class="extype" name="org.apache">apache</a>.<a href="../../package.html" class="extype" name="org.apache.gearpump">gearpump</a>.<a href="../package.html" class="extype" name="org.apache.gearpump.streaming">streaming</a></p> |
| <h1>source</h1><span class="permalink"> |
| <a href="../../../../../index.html#org.apache.gearpump.streaming.source.package" title="Permalink" target="_top"> |
| <img src="../../../../../lib/permalink.png" alt="Permalink" /> |
| </a> |
| </span> |
| </div> |
| |
| <h4 id="signature" class="signature"> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">package</span> |
| </span> |
| <span class="symbol"> |
| <span class="name">source</span> |
| </span> |
| </h4> |
| |
| <div id="comment" class="fullcommenttop"></div> |
| |
| |
| <div id="mbrsel"> |
| <div id="textfilter"><span class="pre"></span><span class="input"><input id="mbrsel-input" type="text" accesskey="/" /></span><span class="post"></span></div> |
| |
| |
| <div id="visbl"> |
| <span class="filtertype">Visibility</span> |
| <ol><li class="public in"><span>Public</span></li><li class="all out"><span>All</span></li></ol> |
| </div> |
| </div> |
| |
| <div id="template"> |
| <div id="allMembers"> |
| |
| |
| <div id="types" class="types members"> |
| <h3>Type Members</h3> |
| <ol><li name="org.apache.gearpump.streaming.source.DataSource" visbl="pub" data-isabs="true" fullComment="yes" group="Ungrouped"> |
| <a id="DataSourceextendsSerializable"></a> |
| <a id="DataSource:DataSource"></a> |
| <h4 class="signature"> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">trait</span> |
| </span> |
| <span class="symbol"> |
| <a href="DataSource.html"><span class="name">DataSource</span></a><span class="result"> extends <span class="extype" name="java.io.Serializable">Serializable</span></span> |
| </span> |
| </h4><span class="permalink"> |
| <a href="../../../../../index.html#org.apache.gearpump.streaming.source.package@DataSourceextendsSerializable" title="Permalink" target="_top"> |
| <img src="../../../../../lib/permalink.png" alt="Permalink" /> |
| </a> |
| </span> |
| <p class="shortcomment cmt">Interface to implement custom source where data is read into the system.</p><div class="fullcomment"><div class="comment cmt"><p>Interface to implement custom source where data is read into the system. |
| a DataSource could be a message queue like kafka or simply data generation source.</p><p>An example would be like</p><pre>GenMsgSource <span class="kw">extends</span> DataSource { |
| |
| <span class="kw">def</span> open(context: TaskContext, startTime: TimeStamp): <span class="std">Unit</span> = {} |
| |
| <span class="kw">def</span> read(context: TaskContext): Message = { |
| Message(<span class="lit">"message"</span>) |
| } |
| |
| <span class="kw">def</span> close(): <span class="std">Unit</span> = {} |
| }</pre><p>subclass is required to be serializable |
| </p></div></div> |
| </li><li name="org.apache.gearpump.streaming.source.DataSourceTask" visbl="pub" data-isabs="false" fullComment="yes" group="Ungrouped"> |
| <a id="DataSourceTask[IN,OUT]extendsTask"></a> |
| <a id="DataSourceTask[IN,OUT]:DataSourceTask[IN,OUT]"></a> |
| <h4 class="signature"> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">class</span> |
| </span> |
| <span class="symbol"> |
| <a href="DataSourceTask.html"><span class="name">DataSourceTask</span></a><span class="tparams">[<span name="IN">IN</span>, <span name="OUT">OUT</span>]</span><span class="result"> extends <a href="../task/Task.html" class="extype" name="org.apache.gearpump.streaming.task.Task">Task</a></span> |
| </span> |
| </h4><span class="permalink"> |
| <a href="../../../../../index.html#org.apache.gearpump.streaming.source.package@DataSourceTask[IN,OUT]extendsTask" title="Permalink" target="_top"> |
| <img src="../../../../../lib/permalink.png" alt="Permalink" /> |
| </a> |
| </span> |
| <p class="shortcomment cmt">Default Task container for <a href="DataSource.html" class="extype" name="org.apache.gearpump.streaming.source.DataSource">org.apache.gearpump.streaming.source.DataSource</a> that |
| reads from DataSource in batch |
| See <a href="DataSourceProcessor$.html" class="extype" name="org.apache.gearpump.streaming.source.DataSourceProcessor">org.apache.gearpump.streaming.source.DataSourceProcessor</a> for its usage</p><div class="fullcomment"><div class="comment cmt"><p>Default Task container for <a href="DataSource.html" class="extype" name="org.apache.gearpump.streaming.source.DataSource">org.apache.gearpump.streaming.source.DataSource</a> that |
| reads from DataSource in batch |
| See <a href="DataSourceProcessor$.html" class="extype" name="org.apache.gearpump.streaming.source.DataSourceProcessor">org.apache.gearpump.streaming.source.DataSourceProcessor</a> for its usage</p><p>DataSourceTask calls:</p><ul><li><code>DataSource.open()</code> in <code>onStart</code> and pass in |
| <a href="../task/TaskContext.html" class="extype" name="org.apache.gearpump.streaming.task.TaskContext">org.apache.gearpump.streaming.task.TaskContext</a> |
| and application start time</li><li><code>DataSource.read()</code> in each <code>onNext</code>, which reads a batch of messages</li><li><code>DataSource.close()</code> in <code>onStop</code> |
| </li></ul></div></div> |
| </li><li name="org.apache.gearpump.streaming.source.Watermark" visbl="pub" data-isabs="false" fullComment="no" group="Ungrouped"> |
| <a id="WatermarkextendsProductwithSerializable"></a> |
| <a id="Watermark:Watermark"></a> |
| <h4 class="signature"> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">case class</span> |
| </span> |
| <span class="symbol"> |
| <a href="Watermark.html"><span class="name">Watermark</span></a><span class="params">(<span name="instant">instant: <span class="extype" name="java.time.Instant">Instant</span></span>)</span><span class="result"> extends <span class="extype" name="scala.Product">Product</span> with <span class="extype" name="scala.Serializable">Serializable</span></span> |
| </span> |
| </h4><span class="permalink"> |
| <a href="../../../../../index.html#org.apache.gearpump.streaming.source.package@WatermarkextendsProductwithSerializable" title="Permalink" target="_top"> |
| <img src="../../../../../lib/permalink.png" alt="Permalink" /> |
| </a> |
| </span> |
| <p class="shortcomment cmt">message used by source task to report source watermark.</p> |
| </li></ol> |
| </div> |
| |
| |
| |
| <div id="values" class="values members"> |
| <h3>Value Members</h3> |
| <ol><li name="org.apache.gearpump.streaming.source.DataSourceConfig" visbl="pub" data-isabs="false" fullComment="no" group="Ungrouped"> |
| <a id="DataSourceConfig"></a> |
| <a id="DataSourceConfig:DataSourceConfig"></a> |
| <h4 class="signature"> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">object</span> |
| </span> |
| <span class="symbol"> |
| <a href="DataSourceConfig$.html"><span class="name">DataSourceConfig</span></a> |
| </span> |
| </h4><span class="permalink"> |
| <a href="../../../../../index.html#org.apache.gearpump.streaming.source.package@DataSourceConfig" title="Permalink" target="_top"> |
| <img src="../../../../../lib/permalink.png" alt="Permalink" /> |
| </a> |
| </span> |
| |
| </li><li name="org.apache.gearpump.streaming.source.DataSourceProcessor" visbl="pub" data-isabs="false" fullComment="yes" group="Ungrouped"> |
| <a id="DataSourceProcessor"></a> |
| <a id="DataSourceProcessor:DataSourceProcessor"></a> |
| <h4 class="signature"> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">object</span> |
| </span> |
| <span class="symbol"> |
| <a href="DataSourceProcessor$.html"><span class="name">DataSourceProcessor</span></a> |
| </span> |
| </h4><span class="permalink"> |
| <a href="../../../../../index.html#org.apache.gearpump.streaming.source.package@DataSourceProcessor" title="Permalink" target="_top"> |
| <img src="../../../../../lib/permalink.png" alt="Permalink" /> |
| </a> |
| </span> |
| <p class="shortcomment cmt">Utility that helps user to create a DAG starting with <a href="DataSourceTask.html" class="extype" name="org.apache.gearpump.streaming.source.DataSourceTask">DataSourceTask</a> |
| user should pass in a <a href="DataSource.html" class="extype" name="org.apache.gearpump.streaming.source.DataSource">DataSource</a></p><div class="fullcomment"><div class="comment cmt"><p>Utility that helps user to create a DAG starting with <a href="DataSourceTask.html" class="extype" name="org.apache.gearpump.streaming.source.DataSourceTask">DataSourceTask</a> |
| user should pass in a <a href="DataSource.html" class="extype" name="org.apache.gearpump.streaming.source.DataSource">DataSource</a></p><p>Here is an example to build a DAG that reads from Kafka source followed by word count</p><pre><span class="kw">val</span> source = <span class="kw">new</span> KafkaSource() |
| <span class="kw">val</span> sourceProcessor = DataSourceProcessor(source, <span class="num">1</span>) |
| <span class="kw">val</span> split = Processor[Split](<span class="num">1</span>) |
| <span class="kw">val</span> sum = Processor[Sum](<span class="num">1</span>) |
| <span class="kw">val</span> dag = sourceProcessor ~> split ~> sum</pre></div></div> |
| </li><li name="org.apache.gearpump.streaming.source.Watermark" visbl="pub" data-isabs="false" fullComment="no" group="Ungrouped"> |
| <a id="Watermark"></a> |
| <a id="Watermark:Watermark"></a> |
| <h4 class="signature"> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">object</span> |
| </span> |
| <span class="symbol"> |
| <a href="Watermark$.html"><span class="name">Watermark</span></a><span class="result"> extends <span class="extype" name="scala.Serializable">Serializable</span></span> |
| </span> |
| </h4><span class="permalink"> |
| <a href="../../../../../index.html#org.apache.gearpump.streaming.source.package@Watermark" title="Permalink" target="_top"> |
| <img src="../../../../../lib/permalink.png" alt="Permalink" /> |
| </a> |
| </span> |
| |
| </li></ol> |
| </div> |
| |
| |
| |
| |
| </div> |
| |
| <div id="inheritedMembers"> |
| |
| |
| </div> |
| |
| <div id="groupedMembers"> |
| <div class="group" name="Ungrouped"> |
| <h3>Ungrouped</h3> |
| |
| </div> |
| </div> |
| |
| </div> |
| |
| <div id="tooltip"></div> |
| |
| <div id="footer"> </div> |
| |
| |
| </body> |
| </html> |
