Google Git
Sign in
apache / flink-web / 0f9478d1de8c2317e67a6a63df5961eba5bbc8db / . / content / 2015 / 03 / 02 / february-2015-in-the-flink-community / index.html
blob: 3b1512141fae6f3cdf2a653f02b236bf8fa8669a [file] [log] [blame]
<!DOCTYPE html>
<html lang="en" dir=ZgotmplZ>
<head>
<link rel="stylesheet" href="/bootstrap/css/bootstrap.min.css">
<script src="/bootstrap/js/bootstrap.bundle.min.js"></script>
<link rel="stylesheet" type="text/css" href="/font-awesome/css/font-awesome.min.css">
<script src="/js/anchor.min.js"></script>
<script src="/js/flink.js"></script>
<link rel="canonical" href="https://flink.apache.org/2015/03/02/february-2015-in-the-flink-community/">
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="description" content="February might be the shortest month of the year, but this does not mean that the Flink community has not been busy adding features to the system and fixing bugs. Here’s a rundown of the activity in the Flink community last month.
0.8.1 release # Flink 0.8.1 was released. This bugfixing release resolves a total of 22 issues.
New committer # Max Michels has been voted a committer by the Flink PMC.">
<meta name="theme-color" content="#FFFFFF"><meta property="og:title" content="February 2015 in the Flink community" />
<meta property="og:description" content="February might be the shortest month of the year, but this does not mean that the Flink community has not been busy adding features to the system and fixing bugs. Here’s a rundown of the activity in the Flink community last month.
0.8.1 release # Flink 0.8.1 was released. This bugfixing release resolves a total of 22 issues.
New committer # Max Michels has been voted a committer by the Flink PMC." />
<meta property="og:type" content="article" />
<meta property="og:url" content="https://flink.apache.org/2015/03/02/february-2015-in-the-flink-community/" /><meta property="article:section" content="posts" />
<meta property="article:published_time" content="2015-03-02T10:00:00+00:00" />
<meta property="article:modified_time" content="2015-03-02T10:00:00+00:00" />
<title>February 2015 in the Flink community | Apache Flink</title>
<link rel="manifest" href="/manifest.json">
<link rel="icon" href="/favicon.png" type="image/x-icon">
<link rel="stylesheet" href="/book.min.22eceb4d17baa9cdc0f57345edd6f215a40474022dfee39b63befb5fb3c596b5.css" integrity="sha256-IuzrTRe6qc3A9XNF7dbyFaQEdAIt/uObY777X7PFlrU=">
<script defer src="/en.search.min.2698f0d1b683dae4d6cb071668b310a55ebcf1c48d11410a015a51d90105b53e.js" integrity="sha256-Jpjw0baD2uTWywcWaLMQpV688cSNEUEKAVpR2QEFtT4="></script>
<!--
Made with Book Theme
https://github.com/alex-shpak/hugo-book
-->
<meta name="generator" content="Hugo 0.124.1">
<script>
var _paq = window._paq = window._paq || [];
_paq.push(['disableCookies']);
_paq.push(["setDomains", ["*.flink.apache.org","*.nightlies.apache.org/flink"]]);
_paq.push(['trackPageView']);
_paq.push(['enableLinkTracking']);
(function() {
var u="//analytics.apache.org/";
_paq.push(['setTrackerUrl', u+'matomo.php']);
_paq.push(['setSiteId', '1']);
var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0];
g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
})();
</script>
</head>
<body dir=ZgotmplZ>
<header>
<nav class="navbar navbar-expand-xl">
<div class="container-fluid">
<a class="navbar-brand" href="/">
<img src="/img/logo/png/100/flink_squirrel_100_color.png" alt="Apache Flink" height="47" width="47" class="d-inline-block align-text-middle">
<span>Apache Flink</span>
</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarSupportedContent" aria-controls="navbarSupportedContent" aria-expanded="false" aria-label="Toggle navigation">
<i class="fa fa-bars navbar-toggler-icon"></i>
</button>
<div class="collapse navbar-collapse" id="navbarSupportedContent">
<ul class="navbar-nav">
<li class="nav-item dropdown">
<a class="nav-link dropdown-toggle" href="#" role="button" data-bs-toggle="dropdown" aria-expanded="false">About</a>
<ul class="dropdown-menu">
<li>
<a class="dropdown-item" href="/what-is-flink/flink-architecture/">Architecture</a>
</li>
<li>
<a class="dropdown-item" href="/what-is-flink/flink-applications/">Applications</a>
</li>
<li>
<a class="dropdown-item" href="/what-is-flink/flink-operations/">Operations</a>
</li>
<li>
<a class="dropdown-item" href="/what-is-flink/use-cases/">Use Cases</a>
</li>
<li>
<a class="dropdown-item" href="/what-is-flink/powered-by/">Powered By</a>
</li>
<li>
<a class="dropdown-item" href="/what-is-flink/roadmap/">Roadmap</a>
</li>
<li>
<a class="dropdown-item" href="/what-is-flink/community/">Community & Project Info</a>
</li>
<li>
<a class="dropdown-item" href="/what-is-flink/security/">Security</a>
</li>
<li>
<a class="dropdown-item" href="/what-is-flink/special-thanks/">Special Thanks</a>
</li>
</ul>
</li>
<li class="nav-item dropdown">
<a class="nav-link dropdown-toggle" href="#" role="button" data-bs-toggle="dropdown" aria-expanded="false">Getting Started</a>
<ul class="dropdown-menu">
<li>
<a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-docs-stable/docs/try-flink/local_installation/">With Flink<i class="link fa fa-external-link title" aria-hidden="true"></i>
</a>
</li>
<li>
<a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-stable/docs/try-flink-kubernetes-operator/quick-start/">With Flink Kubernetes Operator<i class="link fa fa-external-link title" aria-hidden="true"></i>
</a>
</li>
<li>
<a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/get-started/introduction/">With Flink CDC<i class="link fa fa-external-link title" aria-hidden="true"></i>
</a>
</li>
<li>
<a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-ml-docs-stable/docs/try-flink-ml/quick-start/">With Flink ML<i class="link fa fa-external-link title" aria-hidden="true"></i>
</a>
</li>
<li>
<a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-statefun-docs-stable/getting-started/project-setup.html">With Flink Stateful Functions<i class="link fa fa-external-link title" aria-hidden="true"></i>
</a>
</li>
<li>
<a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-docs-stable/docs/learn-flink/overview/">Training Course<i class="link fa fa-external-link title" aria-hidden="true"></i>
</a>
</li>
</ul>
</li>
<li class="nav-item dropdown">
<a class="nav-link dropdown-toggle" href="#" role="button" data-bs-toggle="dropdown" aria-expanded="false">Documentation</a>
<ul class="dropdown-menu">
<li>
<a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-docs-stable/">Flink 1.19 (stable)<i class="link fa fa-external-link title" aria-hidden="true"></i>
</a>
</li>
<li>
<a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-docs-master/">Flink Master (snapshot)<i class="link fa fa-external-link title" aria-hidden="true"></i>
</a>
</li>
<li>
<a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-stable/">Kubernetes Operator 1.8 (latest)<i class="link fa fa-external-link title" aria-hidden="true"></i>
</a>
</li>
<li>
<a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main">Kubernetes Operator Main (snapshot)<i class="link fa fa-external-link title" aria-hidden="true"></i>
</a>
</li>
<li>
<a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-cdc-docs-stable">CDC 3.0 (stable)<i class="link fa fa-external-link title" aria-hidden="true"></i>
</a>
</li>
<li>
<a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-cdc-docs-master">CDC Master (snapshot)<i class="link fa fa-external-link title" aria-hidden="true"></i>
</a>
</li>
<li>
<a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-ml-docs-stable/">ML 2.3 (stable)<i class="link fa fa-external-link title" aria-hidden="true"></i>
</a>
</li>
<li>
<a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-ml-docs-master">ML Master (snapshot)<i class="link fa fa-external-link title" aria-hidden="true"></i>
</a>
</li>
<li>
<a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-statefun-docs-stable/">Stateful Functions 3.3 (stable)<i class="link fa fa-external-link title" aria-hidden="true"></i>
</a>
</li>
<li>
<a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-statefun-docs-master">Stateful Functions Master (snapshot)<i class="link fa fa-external-link title" aria-hidden="true"></i>
</a>
</li>
</ul>
</li>
<li class="nav-item dropdown">
<a class="nav-link dropdown-toggle" href="#" role="button" data-bs-toggle="dropdown" aria-expanded="false">How to Contribute</a>
<ul class="dropdown-menu">
<li>
<a class="dropdown-item" href="/how-to-contribute/overview/">Overview</a>
</li>
<li>
<a class="dropdown-item" href="/how-to-contribute/contribute-code/">Contribute Code</a>
</li>
<li>
<a class="dropdown-item" href="/how-to-contribute/reviewing-prs/">Review Pull Requests</a>
</li>
<li>
<a class="dropdown-item" href="/how-to-contribute/code-style-and-quality-preamble/">Code Style and Quality Guide</a>
</li>
<li>
<a class="dropdown-item" href="/how-to-contribute/contribute-documentation/">Contribute Documentation</a>
</li>
<li>
<a class="dropdown-item" href="/how-to-contribute/documentation-style-guide/">Documentation Style Guide</a>
</li>
<li>
<a class="dropdown-item" href="/how-to-contribute/improve-website/">Contribute to the Website</a>
</li>
<li>
<a class="dropdown-item" href="/how-to-contribute/getting-help/">Getting Help</a>
</li>
</ul>
</li>
<li class="nav-item">
<a class="nav-link" href="/posts/">Flink Blog</a>
</li>
<li class="nav-item">
<a class="nav-link" href="/downloads/">Downloads</a>
</li>
</ul>
<div class="book-search">
<div class="book-search-spinner hidden">
<i class="fa fa-refresh fa-spin"></i>
</div>
<form class="search-bar d-flex" onsubmit="return false;"su>
<input type="text" id="book-search-input" placeholder="Search" aria-label="Search" maxlength="64" data-hotkeys="s/">
<i class="fa fa-search search"></i>
<i class="fa fa-circle-o-notch fa-spin spinner"></i>
</form>
<div class="book-search-spinner hidden"></div>
<ul id="book-search-results"></ul>
</div>
</div>
</div>
</nav>
<div class="navbar-clearfix"></div>
</header>
<main class="flex">
<section class="container book-page">
<article class="markdown">
<h1>
<a href="/2015/03/02/february-2015-in-the-flink-community/">February 2015 in the Flink community</a>
</h1>
March 2, 2015 -
<p><p>February might be the shortest month of the year, but this does not
mean that the Flink community has not been busy adding features to the
system and fixing bugs. Here’s a rundown of the activity in the Flink
community last month.</p>
<h3 id="081-release">
0.8.1 release
<a class="anchor" href="#081-release">#</a>
</h3>
<p>Flink 0.8.1 was released. This bugfixing release resolves a total of 22 issues.</p>
<h3 id="new-committer">
New committer
<a class="anchor" href="#new-committer">#</a>
</h3>
<p><a href="https://github.com/mxm">Max Michels</a> has been voted a committer by the Flink PMC.</p>
<h3 id="flink-adapter-for-apache-samoa">
Flink adapter for Apache SAMOA
<a class="anchor" href="#flink-adapter-for-apache-samoa">#</a>
</h3>
<p><a href="http://samoa.incubator.apache.org">Apache SAMOA (incubating)</a> is a
distributed streaming machine learning (ML) framework with a
programming abstraction for distributed streaming ML algorithms. SAMOA
runs on a variety of backend engines, currently Apache Storm and
Apache S4. A <a href="https://github.com/apache/incubator-samoa/pull/11">pull
request</a> is
available at the SAMOA repository that adds a Flink adapter for SAMOA.</p>
<h3 id="easy-flink-deployment-on-google-compute-cloud">
Easy Flink deployment on Google Compute Cloud
<a class="anchor" href="#easy-flink-deployment-on-google-compute-cloud">#</a>
</h3>
<p>Flink is now integrated in bdutil, Google’s open source tool for
creating and configuring (Hadoop) clusters in Google Compute
Engine. Deployment of Flink clusters in now supported starting with
<a href="https://groups.google.com/forum/#!topic/gcp-hadoop-announce/uVJ_6y9cGKM">bdutil
1.2.0</a>.</p>
<h3 id="flink-on-the-web">
Flink on the Web
<a class="anchor" href="#flink-on-the-web">#</a>
</h3>
<p>A new blog post on <a href="http://flink.apache.org/news/2015/02/09/streaming-example.html">Flink
Streaming</a>
was published at the blog. Flink was mentioned in several articles on
the web. Here are some examples:</p>
<ul>
<li>
<p><a href="http://dataconomy.com/how-flink-became-an-apache-top-level-project/">How Flink became an Apache Top-Level Project</a></p>
</li>
<li>
<p><a href="https://www.linkedin.com/pulse/stale-synchronous-parallelism-new-frontier-apache-flink-nam-luc-tran?utm_content=buffer461af&amp;utm_medium=social&amp;utm_source=linkedin.com&amp;utm_campaign=buffer">Stale Synchronous Parallelism: The new frontier for Apache Flink?</a></p>
</li>
<li>
<p><a href="http://www.hadoopsphere.com/2015/02/distributed-data-processing-with-apache.html">Distributed data processing with Apache Flink</a></p>
</li>
<li>
<p><a href="http://www.hadoopsphere.com/2015/02/ciao-latency-hallo-speed.html">Ciao latency, hello speed</a></p>
</li>
</ul>
<h2 id="in-the-flink-master">
In the Flink master
<a class="anchor" href="#in-the-flink-master">#</a>
</h2>
<p>The following features have been now merged in Flink’s master repository.</p>
<h3 id="gelly">
Gelly
<a class="anchor" href="#gelly">#</a>
</h3>
<p>Gelly, Flink’s Graph API allows users to manipulate graph-shaped data
directly. Here’s for example a calculation of shortest paths in a
graph:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">Graph</span><span class="o">&lt;</span><span class="n">Long</span><span class="p">,</span><span class="w"> </span><span class="n">Double</span><span class="p">,</span><span class="w"> </span><span class="n">Double</span><span class="o">&gt;</span><span class="w"> </span><span class="n">graph</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Graph</span><span class="p">.</span><span class="na">fromDataSet</span><span class="p">(</span><span class="n">vertices</span><span class="p">,</span><span class="w"> </span><span class="n">edges</span><span class="p">,</span><span class="w"> </span><span class="n">env</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="n">DataSet</span><span class="o">&lt;</span><span class="n">Vertex</span><span class="o">&lt;</span><span class="n">Long</span><span class="p">,</span><span class="w"> </span><span class="n">Double</span><span class="o">&gt;&gt;</span><span class="w"> </span><span class="n">singleSourceShortestPaths</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">graph</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="p">.</span><span class="na">run</span><span class="p">(</span><span class="k">new</span><span class="w"> </span><span class="n">SingleSourceShortestPaths</span><span class="o">&lt;</span><span class="n">Long</span><span class="o">&gt;</span><span class="p">(</span><span class="n">srcVertexId</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="n">maxIterations</span><span class="p">)).</span><span class="na">getVertices</span><span class="p">();</span></span></span></code></pre></div>
<p>See more Gelly examples
<a href="https://github.com/apache/flink/tree/master/flink-libraries/flink-gelly-examples">here</a>.</p>
<h3 id="flink-expressions">
Flink Expressions
<a class="anchor" href="#flink-expressions">#</a>
</h3>
<p>The newly merged
<a href="https://github.com/apache/flink/tree/master/flink-libraries/flink-table">flink-table</a>
module is the first step in Flink’s roadmap towards logical queries
and SQL support. Here’s a preview on how you can read two CSV file,
assign a logical schema to, and apply transformations like filters and
joins using logical attributes rather than physical data types.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">val</span> <span class="n">customers</span> <span class="k">=</span> <span class="n">getCustomerDataSet</span><span class="o">(</span><span class="n">env</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="n">as</span><span class="o">(</span>&#39;id<span class="o">,</span> &#39;mktSegment<span class="o">)</span>
</span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="n">filter</span><span class="o">(</span> &#39;mktSegment <span class="o">===</span> <span class="s">&#34;AUTOMOBILE&#34;</span> <span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">orders</span> <span class="k">=</span> <span class="n">getOrdersDataSet</span><span class="o">(</span><span class="n">env</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="n">filter</span><span class="o">(</span> <span class="n">o</span> <span class="k">=&gt;</span> <span class="n">dateFormat</span><span class="o">.</span><span class="n">parse</span><span class="o">(</span><span class="n">o</span><span class="o">.</span><span class="n">orderDate</span><span class="o">).</span><span class="n">before</span><span class="o">(</span><span class="n">date</span><span class="o">)</span> <span class="o">)</span>
</span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="n">as</span><span class="o">(</span>&#39;orderId<span class="o">,</span> &#39;custId<span class="o">,</span> &#39;orderDate<span class="o">,</span> &#39;shipPrio<span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">items</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl"> <span class="n">orders</span><span class="o">.</span><span class="n">join</span><span class="o">(</span><span class="n">customers</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="n">where</span><span class="o">(</span>&#39;custId <span class="o">===</span> &#39;id<span class="o">)</span>
</span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="n">select</span><span class="o">(</span>&#39;orderId<span class="o">,</span> &#39;orderDate<span class="o">,</span> &#39;shipPrio<span class="o">)</span></span></span></code></pre></div>
<h3 id="access-to-hcatalog-tables">
Access to HCatalog tables
<a class="anchor" href="#access-to-hcatalog-tables">#</a>
</h3>
<p>With the <a href="https://github.com/apache/flink/tree/master/flink-batch-connectors/flink-hcatalog">flink-hcatalog
module</a>,
you can now conveniently access HCatalog/Hive tables. The module
supports projection (selection and order of fields) and partition
filters.</p>
<h3 id="access-to-secured-yarn-clustershdfs">
Access to secured YARN clusters/HDFS.
<a class="anchor" href="#access-to-secured-yarn-clustershdfs">#</a>
</h3>
<p>With this change users can access Kerberos secured YARN (and HDFS)
Hadoop clusters. Also, basic support for accessing secured HDFS with
a standalone Flink setup is now available.</p>
</p>
</article>
<div class="edit-this-page">
<p>
<a href="https://cwiki.apache.org/confluence/display/FLINK/Flink+Translation+Specifications">Want to contribute translation?</a>
</p>
<p>
<a href="//github.com/apache/flink-web/edit/asf-site/docs/content/posts/2015-03-02-february-2015-in-flink.md">
Edit This Page<i class="fa fa-edit fa-fw"></i>
</a>
</p>
</div>
</section>
<aside class="book-toc">
<nav id="TableOfContents"><h3>On This Page <a href="javascript:void(0)" class="toc" onclick="collapseToc()"><i class="fa fa-times" aria-hidden="true"></i></a></h3>
<ul>
<li>
<ul>
<li>
<ul>
<li><a href="#081-release">0.8.1 release</a></li>
<li><a href="#new-committer">New committer</a></li>
<li><a href="#flink-adapter-for-apache-samoa">Flink adapter for Apache SAMOA</a></li>
<li><a href="#easy-flink-deployment-on-google-compute-cloud">Easy Flink deployment on Google Compute Cloud</a></li>
<li><a href="#flink-on-the-web">Flink on the Web</a></li>
</ul>
</li>
<li><a href="#in-the-flink-master">In the Flink master</a>
<ul>
<li><a href="#gelly">Gelly</a></li>
<li><a href="#flink-expressions">Flink Expressions</a></li>
<li><a href="#access-to-hcatalog-tables">Access to HCatalog tables</a></li>
<li><a href="#access-to-secured-yarn-clustershdfs">Access to secured YARN clusters/HDFS.</a></li>
</ul>
</li>
</ul>
</li>
</ul>
</nav>
</aside>
<aside class="expand-toc hidden">
<a class="toc" onclick="expandToc()" href="javascript:void(0)">
<i class="fa fa-bars" aria-hidden="true"></i>
</a>
</aside>
</main>
<footer>
<div class="separator"></div>
<div class="panels">
<div class="wrapper">
<div class="panel">
<ul>
<li>
<a href="https://flink-packages.org/">flink-packages.org</a>
</li>
<li>
<a href="https://www.apache.org/">Apache Software Foundation</a>
</li>
<li>
<a href="https://www.apache.org/licenses/">License</a>
</li>
<li>
<a href="/zh/">
<i class="fa fa-globe" aria-hidden="true"></i>&nbsp;中文版
</a>
</li>
</ul>
</div>
<div class="panel">
<ul>
<li>
<a href="/what-is-flink/security">Security</a-->
</li>
<li>
<a href="https://www.apache.org/foundation/sponsorship.html">Donate</a>
</li>
<li>
<a href="https://www.apache.org/foundation/thanks.html">Thanks</a>
</li>
</ul>
</div>
<div class="panel icons">
<div>
<a href="/posts">
<div class="icon flink-blog-icon"></div>
<span>Flink blog</span>
</a>
</div>
<div>
<a href="https://github.com/apache/flink">
<div class="icon flink-github-icon"></div>
<span>Github</span>
</a>
</div>
<div>
<a href="https://twitter.com/apacheflink">
<div class="icon flink-twitter-icon"></div>
<span>Twitter</span>
</a>
</div>
</div>
</div>
</div>
<hr/>
<div class="container disclaimer">
<p>The contents of this website are © 2024 Apache Software Foundation under the terms of the Apache License v2. Apache Flink, Flink, and the Flink logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.</p>
</div>
</footer>
</body>
</html>