| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8"> |
| <meta http-equiv="X-UA-Compatible" content="IE=edge"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags --> |
| <title>Apache Flink: Getting Help</title> |
| <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon"> |
| <link rel="icon" href="/favicon.ico" type="image/x-icon"> |
| |
| <!-- Bootstrap --> |
| <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.4.1/css/bootstrap.min.css"> |
| <link rel="stylesheet" href="/css/flink.css"> |
| <link rel="stylesheet" href="/css/syntax.css"> |
| |
| <!-- Blog RSS feed --> |
| <link href="/blog/feed.xml" rel="alternate" type="application/rss+xml" title="Apache Flink Blog: RSS feed" /> |
| |
| <!-- jQuery (necessary for Bootstrap's JavaScript plugins) --> |
| <!-- We need to load Jquery in the header for custom google analytics event tracking--> |
| <script src="/js/jquery.min.js"></script> |
| |
| <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> |
| <!-- WARNING: Respond.js doesn't work if you view the page via file:// --> |
| <!--[if lt IE 9]> |
| <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script> |
| <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script> |
| <![endif]--> |
| </head> |
| <body> |
| |
| |
| <!-- Main content. --> |
| <div class="container"> |
| <div class="row"> |
| |
| |
| <div id="sidebar" class="col-sm-3"> |
| |
| |
| <!-- Top navbar. --> |
| <nav class="navbar navbar-default"> |
| <!-- The logo. --> |
| <div class="navbar-header"> |
| <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1"> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| </button> |
| <div class="navbar-logo"> |
| <a href="/"> |
| <img alt="Apache Flink" src="/img/flink-header-logo.svg" width="147px" height="73px"> |
| </a> |
| </div> |
| </div><!-- /.navbar-header --> |
| |
| <!-- The navigation links. --> |
| <div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1"> |
| <ul class="nav navbar-nav navbar-main"> |
| |
| <!-- First menu section explains visitors what Flink is --> |
| |
| <!-- What is Stream Processing? --> |
| <!-- |
| <li><a href="/streamprocessing1.html">What is Stream Processing?</a></li> |
| --> |
| |
| <!-- What is Flink? --> |
| <li><a href="/flink-architecture.html">What is Apache Flink?</a></li> |
| |
| |
| |
| <!-- What is Stateful Functions? --> |
| |
| <li><a href="/stateful-functions.html">What is Stateful Functions?</a></li> |
| |
| <!-- Use cases --> |
| <li><a href="/usecases.html">Use Cases</a></li> |
| |
| <!-- Powered by --> |
| <li><a href="/poweredby.html">Powered By</a></li> |
| |
| |
| |
| <!-- Second menu section aims to support Flink users --> |
| |
| <!-- Downloads --> |
| <li><a href="/downloads.html">Downloads</a></li> |
| |
| <!-- Getting Started --> |
| <li class="dropdown"> |
| <a class="dropdown-toggle" data-toggle="dropdown" href="#">Getting Started<span class="caret"></span></a> |
| <ul class="dropdown-menu"> |
| <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/getting-started/index.html" target="_blank">With Flink <small><span class="glyphicon glyphicon-new-window"></span></small></a></li> |
| <li><a href="https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/getting-started/project-setup.html" target="_blank">With Flink Stateful Functions <small><span class="glyphicon glyphicon-new-window"></span></small></a></li> |
| <li><a href="/training.html">Training Course</a></li> |
| </ul> |
| </li> |
| |
| <!-- Documentation --> |
| <li class="dropdown"> |
| <a class="dropdown-toggle" data-toggle="dropdown" href="#">Documentation<span class="caret"></span></a> |
| <ul class="dropdown-menu"> |
| <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11" target="_blank">Flink 1.11 (Latest stable release) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li> |
| <li><a href="https://ci.apache.org/projects/flink/flink-docs-master" target="_blank">Flink Master (Latest Snapshot) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li> |
| <li><a href="https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1" target="_blank">Flink Stateful Functions 2.1 (Latest stable release) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li> |
| <li><a href="https://ci.apache.org/projects/flink/flink-statefun-docs-master" target="_blank">Flink Stateful Functions Master (Latest Snapshot) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li> |
| </ul> |
| </li> |
| |
| <!-- getting help --> |
| <li class="active"><a href="/gettinghelp.html">Getting Help</a></li> |
| |
| <!-- Blog --> |
| <li><a href="/blog/"><b>Flink Blog</b></a></li> |
| |
| |
| <!-- Flink-packages --> |
| <li> |
| <a href="https://flink-packages.org" target="_blank">flink-packages.org <small><span class="glyphicon glyphicon-new-window"></span></small></a> |
| </li> |
| |
| |
| <!-- Third menu section aim to support community and contributors --> |
| |
| <!-- Community --> |
| <li><a href="/community.html">Community & Project Info</a></li> |
| |
| <!-- Roadmap --> |
| <li><a href="/roadmap.html">Roadmap</a></li> |
| |
| <!-- Contribute --> |
| <li><a href="/contributing/how-to-contribute.html">How to Contribute</a></li> |
| |
| |
| <!-- GitHub --> |
| <li> |
| <a href="https://github.com/apache/flink" target="_blank">Flink on GitHub <small><span class="glyphicon glyphicon-new-window"></span></small></a> |
| </li> |
| |
| |
| |
| <!-- Language Switcher --> |
| <li> |
| |
| |
| <a href="/zh/gettinghelp.html">中文版</a> |
| |
| |
| </li> |
| |
| </ul> |
| |
| <ul class="nav navbar-nav navbar-bottom"> |
| <hr /> |
| |
| <!-- Twitter --> |
| <li><a href="https://twitter.com/apacheflink" target="_blank">@ApacheFlink <small><span class="glyphicon glyphicon-new-window"></span></small></a></li> |
| |
| <!-- Visualizer --> |
| <li class=" hidden-md hidden-sm"><a href="/visualizer/" target="_blank">Plan Visualizer <small><span class="glyphicon glyphicon-new-window"></span></small></a></li> |
| |
| <hr /> |
| |
| <li><a href="https://apache.org" target="_blank">Apache Software Foundation <small><span class="glyphicon glyphicon-new-window"></span></small></a></li> |
| |
| <li> |
| <style> |
| .smalllinks:link { |
| display: inline-block !important; background: none; padding-top: 0px; padding-bottom: 0px; padding-right: 0px; min-width: 75px; |
| } |
| </style> |
| |
| <a class="smalllinks" href="https://www.apache.org/licenses/" target="_blank">License</a> <small><span class="glyphicon glyphicon-new-window"></span></small> |
| |
| <a class="smalllinks" href="https://www.apache.org/security/" target="_blank">Security</a> <small><span class="glyphicon glyphicon-new-window"></span></small> |
| |
| <a class="smalllinks" href="https://www.apache.org/foundation/sponsorship.html" target="_blank">Donate</a> <small><span class="glyphicon glyphicon-new-window"></span></small> |
| |
| <a class="smalllinks" href="https://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a> <small><span class="glyphicon glyphicon-new-window"></span></small> |
| </li> |
| |
| </ul> |
| </div><!-- /.navbar-collapse --> |
| </nav> |
| |
| </div> |
| <div class="col-sm-9"> |
| <div class="row-fluid"> |
| <div class="col-sm-12"> |
| <h1>Getting Help</h1> |
| |
| <hr /> |
| |
| <div class="page-toc"> |
| <ul id="markdown-toc"> |
| <li><a href="#having-a-question" id="markdown-toc-having-a-question">Having a Question?</a> <ul> |
| <li><a href="#user-mailing-list" id="markdown-toc-user-mailing-list">User Mailing List</a></li> |
| <li><a href="#stack-overflow" id="markdown-toc-stack-overflow">Stack Overflow</a></li> |
| </ul> |
| </li> |
| <li><a href="#found-a-bug" id="markdown-toc-found-a-bug">Found a Bug?</a></li> |
| <li><a href="#got-an-error-message" id="markdown-toc-got-an-error-message">Got an Error Message?</a> <ul> |
| <li><a href="#i-have-a-notserializableexception" id="markdown-toc-i-have-a-notserializableexception">I have a NotSerializableException.</a></li> |
| <li><a href="#using-the-scala-api-i-get-an-error-about-implicit-values-and-evidence-parameters" id="markdown-toc-using-the-scala-api-i-get-an-error-about-implicit-values-and-evidence-parameters">Using the Scala API, I get an error about implicit values and evidence parameters.</a></li> |
| <li><a href="#i-see-a-classcastexception-x-cannot-be-cast-to-x" id="markdown-toc-i-see-a-classcastexception-x-cannot-be-cast-to-x">I see a ClassCastException: X cannot be cast to X.</a></li> |
| <li><a href="#i-have-an-abstractmethoderror-or-nosuchfielderror" id="markdown-toc-i-have-an-abstractmethoderror-or-nosuchfielderror">I have an AbstractMethodError or NoSuchFieldError.</a></li> |
| <li><a href="#my-datastream-application-produces-no-output-even-though-events-are-going-in" id="markdown-toc-my-datastream-application-produces-no-output-even-though-events-are-going-in">My DataStream application produces no output, even though events are going in.</a></li> |
| <li><a href="#i-see-an-exception-reporting-insufficient-number-of-network-buffers" id="markdown-toc-i-see-an-exception-reporting-insufficient-number-of-network-buffers">I see an exception reporting “Insufficient number of network buffers”.</a></li> |
| <li><a href="#my-job-fails-with-various-exceptions-from-the-hdfshadoop-code-what-can-i-do" id="markdown-toc-my-job-fails-with-various-exceptions-from-the-hdfshadoop-code-what-can-i-do">My job fails with various exceptions from the HDFS/Hadoop code. What can I do?</a></li> |
| </ul> |
| </li> |
| </ul> |
| |
| </div> |
| |
| <h2 id="having-a-question">Having a Question?</h2> |
| |
| <p>The Apache Flink community answers many user questions every day. You can search for answers and advice in the archives or reach out to the community for help and guidance.</p> |
| |
| <h3 id="user-mailing-list">User Mailing List</h3> |
| |
| <p>Many Flink users, contributors, and committers are subscribed to Flink’s user mailing list. The user mailing list is a very good place to ask for help.</p> |
| |
| <p>Before posting to the mailing list, you can search the mailing list archives for email threads that discuss issues related to yours on the following websites.</p> |
| |
| <ul> |
| <li><a href="https://lists.apache.org/list.html?user@flink.apache.org">Apache Pony Mail Archive</a></li> |
| <li><a href="http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/">Nabble Archive</a></li> |
| </ul> |
| |
| <p>If you’d like to post to the mailing list, you need to</p> |
| |
| <ol> |
| <li>subscribe to the mailing list by sending an email to <code>user-subscribe@flink.apache.org</code>,</li> |
| <li>confirm the subscription by replying to the confirmation email, and</li> |
| <li>send your email to <code>user@flink.apache.org</code>.</li> |
| </ol> |
| |
| <p>Please note that you won’t receive a respose to your mail if you are not subscribed.</p> |
| |
| <h3 id="stack-overflow">Stack Overflow</h3> |
| |
| <p>Many members of the Flink community are active on <a href="https://stackoverflow.com">Stack Overflow</a>. You can search for questions and answers or post your questions using the <a href="https://stackoverflow.com/questions/tagged/apache-flink">[apache-flink]</a> tag.</p> |
| |
| <h2 id="found-a-bug">Found a Bug?</h2> |
| |
| <p>If you observe an unexpected behavior that might be caused by a bug, you can search for reported bugs or file a bug report in <a href="https://issues.apache.org/jira/issues/?jql=project %3D FLINK">Flink’s JIRA</a>.</p> |
| |
| <p>If you are unsure whether the unexpected behavior happend due to a bug or not, please post a question to the <a href="#user-mailing-list">user mailing list</a>.</p> |
| |
| <h2 id="got-an-error-message">Got an Error Message?</h2> |
| |
| <p>Identifying the cause for an error message can be challenging. In the following, we list the most common error messages and explain how to handle them.</p> |
| |
| <h3 id="i-have-a-notserializableexception">I have a NotSerializableException.</h3> |
| |
| <p>Flink uses Java serialization to distribute copies of the application logic (the functions and operations you implement, |
| as well as the program configuration, etc.) to the parallel worker processes. |
| Because of that, all functions that you pass to the API must be serializable, as defined by |
| <a href="http://docs.oracle.com/javase/8/docs/api/java/io/Serializable.html">java.io.Serializable</a>.</p> |
| |
| <p>If your function is an anonymous inner class, consider the following:</p> |
| |
| <ul> |
| <li>make the function a standalone class, or a static inner class.</li> |
| <li>use a Java 8 lambda function.</li> |
| </ul> |
| |
| <p>Is your function is already a static class, check the fields that you assign when you create |
| an instance of the class. One of the fields most likely holds a non-serializable type.</p> |
| |
| <ul> |
| <li>In Java, use a <code>RichFunction</code> and initialize the problematic fields in the <code>open()</code> method.</li> |
| <li>In Scala, you can often simply use “lazy val” to defer initialization until the distributed execution happens. This may come at a minor performance cost. You can naturally also use a <code>RichFunction</code> in Scala.</li> |
| </ul> |
| |
| <h3 id="using-the-scala-api-i-get-an-error-about-implicit-values-and-evidence-parameters">Using the Scala API, I get an error about implicit values and evidence parameters.</h3> |
| |
| <p>This error means that the implicit value for the type information could not be provided. |
| Make sure that you have an <code>import org.apache.flink.streaming.api.scala._</code> (DataStream API) or an |
| <code>import org.apache.flink.api.scala._</code> (DataSet API) statement in your code.</p> |
| |
| <p>If you are using Flink operations inside functions or classes that take |
| generic parameters, then a TypeInformation must be available for that parameter. |
| This can be achieved by using a context bound:</p> |
| |
| <div class="highlight"><pre><code class="language-scala"><span class="k">def</span> <span class="n">myFunction</span><span class="o">[</span><span class="kt">T:</span> <span class="kt">TypeInformation</span><span class="o">](</span><span class="n">input</span><span class="k">:</span> <span class="kt">DataSet</span><span class="o">[</span><span class="kt">T</span><span class="o">])</span><span class="k">:</span> <span class="kt">DataSet</span><span class="o">[</span><span class="kt">Seq</span><span class="o">[</span><span class="kt">T</span><span class="o">]]</span> <span class="k">=</span> <span class="o">{</span> |
| <span class="n">input</span><span class="o">.</span><span class="n">reduceGroup</span><span class="o">(</span> <span class="n">i</span> <span class="k">=></span> <span class="n">i</span><span class="o">.</span><span class="n">toSeq</span> <span class="o">)</span> |
| <span class="o">}</span></code></pre></div> |
| |
| <p>See <a href="https://ci.apache.org/projects/flink/flink-docs-master/dev/types_serialization.html">Type Extraction and Serialization</a> for |
| an in-depth discussion of how Flink handles types.</p> |
| |
| <h3 id="i-see-a-classcastexception-x-cannot-be-cast-to-x">I see a ClassCastException: X cannot be cast to X.</h3> |
| |
| <p>When you see an exception in the style <code>com.foo.X</code> cannot be cast to <code>com.foo.X</code> (or cannot be assigned to <code>com.foo.X</code>), it means that |
| multiple versions of the class <code>com.foo.X</code> have been loaded by different class loaders, and types of that class are attempted to be assigned to each other.</p> |
| |
| <p>The reason for that can be:</p> |
| |
| <ul> |
| <li> |
| <p>Class duplication through <code>child-first</code> classloading. That is an intended mechanism to allow users to use different versions of the same |
| dependencies that Flink uses. However, if different copies of these classes move between Flink’s core and the user application code, such an exception |
| can occur. To verify that this is the reason, try setting <code>classloader.resolve-order: parent-first</code> in the configuration. |
| If that makes the error disappear, please write to the mailing list to check if that may be a bug.</p> |
| </li> |
| <li> |
| <p>Caching of classes from different execution attempts, for example by utilities like Guava’s Interners, or Avro’s Schema cache. |
| Try to not use interners, or reduce the scope of the interner/cache to make sure a new cache is created whenever a new task |
| execution is started.</p> |
| </li> |
| </ul> |
| |
| <h3 id="i-have-an-abstractmethoderror-or-nosuchfielderror">I have an AbstractMethodError or NoSuchFieldError.</h3> |
| |
| <p>Such errors typically indicate a mix-up in some dependency version. That means a different version of a dependency (a library) |
| is loaded during the execution compared to the version that code was compiled against.</p> |
| |
| <p>From Flink 1.4.0 on, dependencies in your application JAR file may have different versions compared to dependencies used |
| by Flink’s core, or other dependencies in the classpath (for example from Hadoop). That requires <code>child-first</code> classloading |
| to be activated, which is the default.</p> |
| |
| <p>If you see these problems in Flink 1.4+, one of the following may be true:</p> |
| |
| <ul> |
| <li>You have a dependency version conflict within your application code. Make sure all your dependency versions are consistent.</li> |
| <li>You are conflicting with a library that Flink cannot support via <code>child-first</code> classloading. Currently these are the |
| Scala standard library classes, as well as Flink’s own classes, logging APIs, and any Hadoop core classes.</li> |
| </ul> |
| |
| <h3 id="my-datastream-application-produces-no-output-even-though-events-are-going-in">My DataStream application produces no output, even though events are going in.</h3> |
| |
| <p>If your DataStream application uses <em>Event Time</em>, check that your watermarks get updated. If no watermarks are produced, |
| event time windows might never trigger, and the application would produce no results.</p> |
| |
| <p>You can check in Flink’s web UI (watermarks section) whether watermarks are making progress.</p> |
| |
| <h3 id="i-see-an-exception-reporting-insufficient-number-of-network-buffers">I see an exception reporting “Insufficient number of network buffers”.</h3> |
| |
| <p>If you run Flink with a very high parallelism, you may need to increase the number of network buffers.</p> |
| |
| <p>By default, Flink takes 10% of the JVM heap size for network buffers, with a minimum of 64MB and a maximum of 1GB. |
| You can adjust all these values via <code>taskmanager.network.memory.fraction</code>, <code>taskmanager.network.memory.min</code>, and |
| <code>taskmanager.network.memory.max</code>.</p> |
| |
| <p>Please refer to the <a href="https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#configuring-the-network-buffers">Configuration Reference</a> for details.</p> |
| |
| <h3 id="my-job-fails-with-various-exceptions-from-the-hdfshadoop-code-what-can-i-do">My job fails with various exceptions from the HDFS/Hadoop code. What can I do?</h3> |
| |
| <p>The most common cause for that is that the Hadoop version in Flink’s classpath is different than the |
| Hadoop version of the cluster you want to connect to (HDFS / YARN).</p> |
| |
| <p>The easiest way to fix that is to pick a Hadoop-free Flink version and simply export the Hadoop path and |
| classpath from the cluster.</p> |
| |
| |
| </div> |
| </div> |
| |
| </div> |
| </div> |
| |
| <hr /> |
| |
| <div class="row"> |
| <div class="footer text-center col-sm-12"> |
| <p>Copyright © 2014-2019 <a href="http://apache.org">The Apache Software Foundation</a>. All Rights Reserved.</p> |
| <p>Apache Flink, Flink®, Apache®, the squirrel logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation.</p> |
| <p><a href="/privacy-policy.html">Privacy Policy</a> · <a href="/blog/feed.xml">RSS feed</a></p> |
| </div> |
| </div> |
| </div><!-- /.container --> |
| |
| <!-- Include all compiled plugins (below), or include individual files as needed --> |
| <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/js/bootstrap.min.js"></script> |
| <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery.matchHeight/0.7.0/jquery.matchHeight-min.js"></script> |
| <script src="/js/codetabs.js"></script> |
| <script src="/js/stickysidebar.js"></script> |
| |
| <!-- Google Analytics --> |
| <script> |
| (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ |
| (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), |
| m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) |
| })(window,document,'script','//www.google-analytics.com/analytics.js','ga'); |
| |
| ga('create', 'UA-52545728-1', 'auto'); |
| ga('send', 'pageview'); |
| </script> |
| </body> |
| </html> |
