Google Git
Sign in
apache / incubator-retired-gearpump-site / 926356f9c28f27c7f0ec008bb6a77bc2927d7658 / . / releases / 0.6.2 / dev-custom-serializer.html
blob: fc58d88ccd7585b7f2a08cc9643003086e88590a [file] [log] [blame]
<!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]-->
<!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8"> <![endif]-->
<!--[if IE 8]> <html class="no-js lt-ie9"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js"> <!--<![endif]-->
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
<meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1"/>
<title>Gearpump Customize Serializer - GearPump 0.6.2 Documentation</title>
<link rel="stylesheet" href="css/bootstrap-3.3.5.min.css">
<style>
body {
padding-top: 60px;
padding-bottom: 40px;
}
</style>
<link rel="stylesheet" href="css/main.css">
<link rel="stylesheet" href="css/pygments-default.css">
<script src="js/vendor/modernizr-2.6.1-respond-1.1.0.min.js"></script>
</head>
<body>
<!--[if lt IE 7]>
<p class="chromeframe">You are using an outdated browser. <a href="http://browsehappy.com/">Upgrade your browser today</a> or <a href="http://www.google.com/chromeframe/?redirect=true">install Google Chrome Frame</a> to better experience this site.</p>
<![endif]-->
<div class="navbar navbar-inverse navbar-fixed-top" id="topbar">
<div class="container">
<div class="navbar-header">
<a class="navbar-brand" href="/">GearPump
<span class="label label-primary" style="font-size: .6em">0.6.2</span>
</a>
</div>
<div class="collapse navbar-collapse">
<ul class="nav navbar-nav">
<li><a href="index.html">Overview</a></li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Introduction<b class="caret"></b></a>
<ul class="dropdown-menu">
<li><a href="submit-your-1st-application.html">Submit Your 1st Application</a></li>
<li><a href="commandline.html">Client Command Line</a></li>
<li class="divider"></li>
<li><a href="basic-concepts.html">Basic Concepts</a></li>
<li><a href="features.html">Technical Highlights</a></li>
<li><a href="message-delivery.html">Reliable Message Delivery</a></li>
<li><a href="performance-report.html">Performance</a></li>
<li><a href="gearpump-internals.html">Gearpump Internals</a></li>
</ul>
</li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Deploying<b class="caret"></b></a>
<ul class="dropdown-menu">
<li class="dropdown-header">Deployment</li>
<li><a href="deployment-docker.html">Docker</a><li>
<li><a href="deployment-local.html">Local</a><li>
<li><a href="deployment-standalone.html">Standalone</a></li>
<li><a href="deployment-yarn.html">YARN</a></li>
<li class="divider"></li>
<li><a href="deployment-ha.html">High Availability</a></li>
<li><a href="deployment-msg-delivery.html">Reliable Message Delivery</a></li>
<li><a href="deployment-configuration.html">Configuration</a></li>
<li class="divider"></li>
<li><a href="deployment-security.html">Security</a></li>
</ul>
</li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Programming Guide<b class="caret"></b></a>
<ul class="dropdown-menu">
<li><a href="dev-write-1st-app.html">Write Your 1st App</a></li>
<li><a href="dev-custom-serializer.html">Customized Message Passing</a></li>
<li class="divider"></li>
<li><a href="api/scala/index.html">Scala API</a></li>
<li><a href="api/java/index.html">Java API</a></li>
<li><a href="dev-rest-api.html">RESTful API</a></li>
<li class="divider"></li>
<li><a href="dev-connectors.html">Gearpump Connectors</a></li>
<li class="divider"></li>
<li><a href="dev-storm.html">Storm Compatibility</a></li>
<!--
<li><a href="dev-samoa.html">Samoa Compatibility</a></li>
<li class="divider"></li>
<li><a href="dev-iot.html">Gearpump with IoT</a></li>
-->
</ul>
</li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">More<b class="caret"></b></a>
<ul class="dropdown-menu">
<li><a href="how-to-contribute.html">How to Contribute</a></li>
<li><a href="coding-style.html">Coding Style</a></li>
<li class="divider"></li>
<li><a href="faq.html">FAQ</a><li>
<li><a href="about.html">About</a></li>
</ul>
</li>
</ul>
</div>
</div>
</div>
<div class="container" id="content">
<h1 class="title">Gearpump Customize Serializer</h1>
<h4 id="define-custom-message-serializer">Define Custom Message Serializer</h4>
<p>We use library <a href="https://github.com/EsotericSoftware/kryo">kryo</a> and <a href="https://github.com/romix/akka-kryo-serialization">akka-kryo library</a>. If you have special Message type, you can choose to define your own serializer explicitly. If you have not defined your own custom serializer, the system will use Kryo to serialize it at best effort.</p>
<p>When you have determined that you want to define a custom serializer, you can do this in two ways.</p>
<h5 id="system-level-serializer">System Level Serializer</h5>
<p>If the serializer is widely used, you can define a global serializer which is avaiable to all applications(or worker or master) in the system.</p>
<h6 id="step1-you-first-need-to-develop-a-java-library-which-contains-the-custom-serializer-class-here-is-an-example">Step1: you first need to develop a java library which contains the custom serializer class. here is an example:</h6>
<div class="highlight"><pre><code class="language-scala"><span class="k">class</span> <span class="nc">MessageSerializer</span> <span class="k">extends</span> <span class="nc">Serializer</span><span class="o">[</span><span class="kt">Message</span><span class="o">]</span> <span class="o">{</span>
<span class="k">override</span> <span class="k">def</span> <span class="n">write</span><span class="o">(</span><span class="n">kryo</span><span class="k">:</span> <span class="kt">Kryo</span><span class="o">,</span> <span class="n">output</span><span class="k">:</span> <span class="kt">Output</span><span class="o">,</span> <span class="n">obj</span><span class="k">:</span> <span class="kt">Message</span><span class="o">)</span> <span class="k">=</span> <span class="o">{</span>
<span class="n">output</span><span class="o">.</span><span class="n">writeLong</span><span class="o">(</span><span class="n">obj</span><span class="o">.</span><span class="n">timestamp</span><span class="o">)</span>
<span class="n">kryo</span><span class="o">.</span><span class="n">writeClassAndObject</span><span class="o">(</span><span class="n">output</span><span class="o">,</span> <span class="n">obj</span><span class="o">.</span><span class="n">msg</span><span class="o">)</span>
<span class="o">}</span>
<span class="k">override</span> <span class="k">def</span> <span class="n">read</span><span class="o">(</span><span class="n">kryo</span><span class="k">:</span> <span class="kt">Kryo</span><span class="o">,</span> <span class="n">input</span><span class="k">:</span> <span class="kt">Input</span><span class="o">,</span> <span class="n">typ</span><span class="k">:</span> <span class="kt">Class</span><span class="o">[</span><span class="kt">Message</span><span class="o">])</span><span class="k">:</span> <span class="kt">Message</span> <span class="o">=</span> <span class="o">{</span>
<span class="k">var</span> <span class="n">timeStamp</span> <span class="k">=</span> <span class="n">input</span><span class="o">.</span><span class="n">readLong</span><span class="o">()</span>
<span class="k">val</span> <span class="n">msg</span> <span class="k">=</span> <span class="n">kryo</span><span class="o">.</span><span class="n">readClassAndObject</span><span class="o">(</span><span class="n">input</span><span class="o">)</span>
<span class="k">new</span> <span class="nc">Message</span><span class="o">(</span><span class="n">msg</span><span class="o">.</span><span class="n">asInstanceOf</span><span class="o">[</span><span class="kt">java.io.Serializable</span><span class="o">],</span> <span class="n">timeStamp</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span></code></pre></div>
<h6 id="step2-distribute-the-libraries">Step2: Distribute the libraries</h6>
<p>Distribute the jar file to lib/ folder of every Gearpump installation in the cluster.</p>
<h6 id="step3-change-gearconf-on-every-machine-of-the-cluster">Step3: change gear.conf on every machine of the cluster:</h6>
<div class="highlight"><pre><code>gearpump {
serializers {
"io.gearpump.Message" = "your.serializer.class"
}
}
</code></pre></div>
<h5 id="all-set">All set!</h5>
<h4 id="define-application-level-custom-serializer">Define Application level custom serializer</h4>
<p>If all you want is to define an application level serializer, which is only visible to current application AppMaster and Executors(including tasks), you can follow a different approach.</p>
<h6 id="step1-define-your-custom-serializer-class">Step1: Define your custom Serializer class</h6>
<p>You should include the Serializer class in your application jar. Here is an example to define a custom serializer:</p>
<div class="highlight"><pre><code class="language-scala"><span class="k">class</span> <span class="nc">MessageSerializer</span> <span class="k">extends</span> <span class="nc">Serializer</span><span class="o">[</span><span class="kt">Message</span><span class="o">]</span> <span class="o">{</span>
<span class="k">override</span> <span class="k">def</span> <span class="n">write</span><span class="o">(</span><span class="n">kryo</span><span class="k">:</span> <span class="kt">Kryo</span><span class="o">,</span> <span class="n">output</span><span class="k">:</span> <span class="kt">Output</span><span class="o">,</span> <span class="n">obj</span><span class="k">:</span> <span class="kt">Message</span><span class="o">)</span> <span class="k">=</span> <span class="o">{</span>
<span class="n">output</span><span class="o">.</span><span class="n">writeLong</span><span class="o">(</span><span class="n">obj</span><span class="o">.</span><span class="n">timestamp</span><span class="o">)</span>
<span class="n">kryo</span><span class="o">.</span><span class="n">writeClassAndObject</span><span class="o">(</span><span class="n">output</span><span class="o">,</span> <span class="n">obj</span><span class="o">.</span><span class="n">msg</span><span class="o">)</span>
<span class="o">}</span>
<span class="k">override</span> <span class="k">def</span> <span class="n">read</span><span class="o">(</span><span class="n">kryo</span><span class="k">:</span> <span class="kt">Kryo</span><span class="o">,</span> <span class="n">input</span><span class="k">:</span> <span class="kt">Input</span><span class="o">,</span> <span class="n">typ</span><span class="k">:</span> <span class="kt">Class</span><span class="o">[</span><span class="kt">Message</span><span class="o">])</span><span class="k">:</span> <span class="kt">Message</span> <span class="o">=</span> <span class="o">{</span>
<span class="k">var</span> <span class="n">timeStamp</span> <span class="k">=</span> <span class="n">input</span><span class="o">.</span><span class="n">readLong</span><span class="o">()</span>
<span class="k">val</span> <span class="n">msg</span> <span class="k">=</span> <span class="n">kryo</span><span class="o">.</span><span class="n">readClassAndObject</span><span class="o">(</span><span class="n">input</span><span class="o">)</span>
<span class="k">new</span> <span class="nc">Message</span><span class="o">(</span><span class="n">msg</span><span class="o">.</span><span class="n">asInstanceOf</span><span class="o">[</span><span class="kt">java.io.Serializable</span><span class="o">],</span> <span class="n">timeStamp</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span></code></pre></div>
<h6 id="step2-define-a-config-file-to-include-the-custom-serializer-definition-for-example-we-can-create-a-file-called-myconfconf">Step2: Define a config file to include the custom serializer definition. For example, we can create a file called: myconf.conf</h6>
<div class="highlight"><pre><code>### content of myconf.conf
gearpump {
serializers {
"io.gearpump.Message" = "your.serializer.class"
}
}
</code></pre></div>
<h6 id="step3-add-the-conf-into-appdescription">Step3: Add the conf into AppDescription</h6>
<p>Let&#8217;s take WordCount as an example:</p>
<div class="highlight"><pre><code class="language-scala"><span class="k">object</span> <span class="nc">WordCount</span> <span class="k">extends</span> <span class="nc">App</span> <span class="k">with</span> <span class="nc">ArgumentsParser</span> <span class="o">{</span>
<span class="k">private</span> <span class="k">val</span> <span class="nc">LOG</span><span class="k">:</span> <span class="kt">Logger</span> <span class="o">=</span> <span class="nc">LogUtil</span><span class="o">.</span><span class="n">getLogger</span><span class="o">(</span><span class="n">getClass</span><span class="o">)</span>
<span class="k">val</span> <span class="nc">RUN_FOR_EVER</span> <span class="k">=</span> <span class="o">-</span><span class="mi">1</span>
<span class="k">override</span> <span class="k">val</span> <span class="n">options</span><span class="k">:</span> <span class="kt">Array</span><span class="o">[(</span><span class="kt">String</span>, <span class="kt">CLIOption</span><span class="o">[</span><span class="kt">Any</span><span class="o">])]</span> <span class="k">=</span> <span class="nc">Array</span><span class="o">(</span>
<span class="s">&quot;master&quot;</span> <span class="o">-&gt;</span> <span class="nc">CLIOption</span><span class="o">[</span><span class="kt">String</span><span class="o">](</span><span class="s">&quot;&lt;host1:port1,host2:port2,host3:port3&gt;&quot;</span><span class="o">,</span> <span class="n">required</span> <span class="k">=</span> <span class="kc">true</span><span class="o">),</span>
<span class="s">&quot;split&quot;</span> <span class="o">-&gt;</span> <span class="nc">CLIOption</span><span class="o">[</span><span class="kt">Int</span><span class="o">](</span><span class="s">&quot;&lt;how many split tasks&gt;&quot;</span><span class="o">,</span> <span class="n">required</span> <span class="k">=</span> <span class="kc">false</span><span class="o">,</span> <span class="n">defaultValue</span> <span class="k">=</span> <span class="nc">Some</span><span class="o">(</span><span class="mi">4</span><span class="o">)),</span>
<span class="s">&quot;sum&quot;</span> <span class="o">-&gt;</span> <span class="nc">CLIOption</span><span class="o">[</span><span class="kt">Int</span><span class="o">](</span><span class="s">&quot;&lt;how many sum tasks&gt;&quot;</span><span class="o">,</span> <span class="n">required</span> <span class="k">=</span> <span class="kc">false</span><span class="o">,</span> <span class="n">defaultValue</span> <span class="k">=</span> <span class="nc">Some</span><span class="o">(</span><span class="mi">4</span><span class="o">)),</span>
<span class="s">&quot;runseconds&quot;</span><span class="o">-&gt;</span> <span class="nc">CLIOption</span><span class="o">[</span><span class="kt">Int</span><span class="o">](</span><span class="s">&quot;&lt;how long to run this example, set to -1 if run forever&gt;&quot;</span><span class="o">,</span> <span class="n">required</span> <span class="k">=</span> <span class="kc">false</span><span class="o">,</span> <span class="n">defaultValue</span> <span class="k">=</span> <span class="nc">Some</span><span class="o">(</span><span class="mi">60</span><span class="o">))</span>
<span class="o">)</span>
<span class="k">def</span> <span class="n">application</span><span class="o">(</span><span class="n">config</span><span class="k">:</span> <span class="kt">ParseResult</span><span class="o">)</span> <span class="k">:</span> <span class="kt">AppDescription</span> <span class="o">=</span> <span class="o">{</span>
<span class="k">val</span> <span class="n">splitNum</span> <span class="k">=</span> <span class="n">config</span><span class="o">.</span><span class="n">getInt</span><span class="o">(</span><span class="s">&quot;split&quot;</span><span class="o">)</span>
<span class="k">val</span> <span class="n">sumNum</span> <span class="k">=</span> <span class="n">config</span><span class="o">.</span><span class="n">getInt</span><span class="o">(</span><span class="s">&quot;sum&quot;</span><span class="o">)</span>
<span class="k">val</span> <span class="n">partitioner</span> <span class="k">=</span> <span class="k">new</span> <span class="nc">HashPartitioner</span><span class="o">()</span>
<span class="k">val</span> <span class="n">split</span> <span class="k">=</span> <span class="nc">TaskDescription</span><span class="o">(</span><span class="n">classOf</span><span class="o">[</span><span class="kt">Split</span><span class="o">].</span><span class="n">getName</span><span class="o">,</span> <span class="n">splitNum</span><span class="o">)</span>
<span class="k">val</span> <span class="n">sum</span> <span class="k">=</span> <span class="nc">TaskDescription</span><span class="o">(</span><span class="n">classOf</span><span class="o">[</span><span class="kt">Sum</span><span class="o">].</span><span class="n">getName</span><span class="o">,</span> <span class="n">sumNum</span><span class="o">)</span>
<span class="c1">//=======================================</span>
<span class="c1">// Attention!</span>
<span class="c1">//=======================================</span>
<span class="k">val</span> <span class="n">app</span> <span class="k">=</span> <span class="nc">AppDescription</span><span class="o">(</span><span class="s">&quot;wordCount&quot;</span><span class="o">,</span> <span class="nc">UserConfig</span><span class="o">.</span><span class="n">empty</span><span class="o">,</span> <span class="nc">Graph</span><span class="o">(</span><span class="n">split</span> <span class="o">~</span> <span class="n">partitioner</span> <span class="o">~&gt;</span> <span class="n">sum</span><span class="o">),</span>
<span class="nc">ClusterConfigSource</span><span class="o">(</span><span class="s">&quot;/path/to/myconf.conf&quot;</span><span class="o">))</span>
<span class="n">app</span>
<span class="o">}</span>
<span class="k">val</span> <span class="n">config</span> <span class="k">=</span> <span class="n">parse</span><span class="o">(</span><span class="n">args</span><span class="o">)</span>
<span class="k">val</span> <span class="n">context</span> <span class="k">=</span> <span class="nc">ClientContext</span><span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="n">getString</span><span class="o">(</span><span class="s">&quot;master&quot;</span><span class="o">))</span>
<span class="k">implicit</span> <span class="k">val</span> <span class="n">system</span> <span class="k">=</span> <span class="n">context</span><span class="o">.</span><span class="n">system</span>
<span class="k">val</span> <span class="n">appId</span> <span class="k">=</span> <span class="n">context</span><span class="o">.</span><span class="n">submit</span><span class="o">(</span><span class="n">application</span><span class="o">(</span><span class="n">config</span><span class="o">))</span>
<span class="nc">Thread</span><span class="o">.</span><span class="n">sleep</span><span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="n">getInt</span><span class="o">(</span><span class="s">&quot;runseconds&quot;</span><span class="o">)</span> <span class="o">*</span> <span class="mi">1000</span><span class="o">)</span>
<span class="n">context</span><span class="o">.</span><span class="n">shutdown</span><span class="o">(</span><span class="n">appId</span><span class="o">)</span>
<span class="n">context</span><span class="o">.</span><span class="n">close</span><span class="o">()</span>
<span class="o">}</span></code></pre></div>
<p>Maybe you have noticed, we have add a custom config to the Application</p>
<div class="highlight"><pre><code class="language-scala"><span class="c1">//=======================================</span>
<span class="c1">// Attention!</span>
<span class="c1">//=======================================</span>
<span class="k">val</span> <span class="n">app</span> <span class="k">=</span> <span class="nc">AppDescription</span><span class="o">(</span><span class="s">&quot;wordCount&quot;</span><span class="o">,</span> <span class="nc">UserConfig</span><span class="o">.</span><span class="n">empty</span><span class="o">,</span> <span class="nc">Graph</span><span class="o">(</span><span class="n">split</span> <span class="o">~</span> <span class="n">partitioner</span> <span class="o">~&gt;</span> <span class="n">sum</span><span class="o">),</span>
<span class="nc">ClusterConfigSource</span><span class="o">(</span><span class="s">&quot;/path/to/myconf.conf&quot;</span><span class="o">))</span></code></pre></div>
<h6 id="step4-all-set">Step4: All set!</h6>
</div> <!-- /container -->
<script src="js/vendor/jquery-2.1.4.min.js"></script>
<script src="js/vendor/bootstrap-3.3.5.min.js"></script>
<script src="js/vendor/anchor-1.1.1.min.js"></script>
<script src="js/main.js"></script>
<!-- MathJax Section -->
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: { equationNumbers: { autoNumber: "AMS" } }
});
</script>
<script>
// Note that we load MathJax this way to work with local file (file://), HTTP and HTTPS.
// We could use "//cdn.mathjax...", but that won't support "file://".
(function(d, script) {
script = d.createElement('script');
script.type = 'text/javascript';
script.async = true;
script.onload = function(){
MathJax.Hub.Config({
tex2jax: {
inlineMath: [ ["$", "$"], ["\\\\(","\\\\)"] ],
displayMath: [ ["$$","$$"], ["\\[", "\\]"] ],
processEscapes: true,
skipTags: ['script', 'noscript', 'style', 'textarea', 'pre']
}
});
};
script.src = ('https:' == document.location.protocol ? 'https://' : 'http://') +
'cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML';
d.getElementsByTagName('head')[0].appendChild(script);
}(document));
</script>
</body>
</html>