Google Git
Sign in
apache / flink-web / 0f9478d1de8c2317e67a6a63df5961eba5bbc8db / . / content / 2022 / 05 / 05 / announcing-the-release-of-apache-flink-1.15 / index.html
blob: 5ecb5eb67ba69fc46f48b6f700a53d73f1b3900f [file] [log] [blame]
<!DOCTYPE html>
<html lang="en" dir=ZgotmplZ>
<head>
<link rel="stylesheet" href="/bootstrap/css/bootstrap.min.css">
<script src="/bootstrap/js/bootstrap.bundle.min.js"></script>
<link rel="stylesheet" type="text/css" href="/font-awesome/css/font-awesome.min.css">
<script src="/js/anchor.min.js"></script>
<script src="/js/flink.js"></script>
<link rel="canonical" href="https://flink.apache.org/2022/05/05/announcing-the-release-of-apache-flink-1.15/">
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="description" content="Thanks to our well-organized and open community, Apache Flink continues to grow as a technology and remain one of the most active projects in the Apache community. With the release of Flink 1.15, we are proud to announce a number of exciting changes.
One of the main concepts that makes Apache Flink stand out is the unification of batch (aka bounded) and stream (aka unbounded) data processing, which helps reduce the complexity of development.">
<meta name="theme-color" content="#FFFFFF"><meta property="og:title" content="Announcing the Release of Apache Flink 1.15" />
<meta property="og:description" content="Thanks to our well-organized and open community, Apache Flink continues to grow as a technology and remain one of the most active projects in the Apache community. With the release of Flink 1.15, we are proud to announce a number of exciting changes.
One of the main concepts that makes Apache Flink stand out is the unification of batch (aka bounded) and stream (aka unbounded) data processing, which helps reduce the complexity of development." />
<meta property="og:type" content="article" />
<meta property="og:url" content="https://flink.apache.org/2022/05/05/announcing-the-release-of-apache-flink-1.15/" /><meta property="article:section" content="posts" />
<meta property="article:published_time" content="2022-05-05T08:00:00+00:00" />
<meta property="article:modified_time" content="2022-05-05T08:00:00+00:00" />
<title>Announcing the Release of Apache Flink 1.15 | Apache Flink</title>
<link rel="manifest" href="/manifest.json">
<link rel="icon" href="/favicon.png" type="image/x-icon">
<link rel="stylesheet" href="/book.min.22eceb4d17baa9cdc0f57345edd6f215a40474022dfee39b63befb5fb3c596b5.css" integrity="sha256-IuzrTRe6qc3A9XNF7dbyFaQEdAIt/uObY777X7PFlrU=">
<script defer src="/en.search.min.2698f0d1b683dae4d6cb071668b310a55ebcf1c48d11410a015a51d90105b53e.js" integrity="sha256-Jpjw0baD2uTWywcWaLMQpV688cSNEUEKAVpR2QEFtT4="></script>
<!--
Made with Book Theme
https://github.com/alex-shpak/hugo-book
-->
<meta name="generator" content="Hugo 0.124.1">
<script>
var _paq = window._paq = window._paq || [];
_paq.push(['disableCookies']);
_paq.push(["setDomains", ["*.flink.apache.org","*.nightlies.apache.org/flink"]]);
_paq.push(['trackPageView']);
_paq.push(['enableLinkTracking']);
(function() {
var u="//analytics.apache.org/";
_paq.push(['setTrackerUrl', u+'matomo.php']);
_paq.push(['setSiteId', '1']);
var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0];
g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
})();
</script>
</head>
<body dir=ZgotmplZ>
<header>
<nav class="navbar navbar-expand-xl">
<div class="container-fluid">
<a class="navbar-brand" href="/">
<img src="/img/logo/png/100/flink_squirrel_100_color.png" alt="Apache Flink" height="47" width="47" class="d-inline-block align-text-middle">
<span>Apache Flink</span>
</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarSupportedContent" aria-controls="navbarSupportedContent" aria-expanded="false" aria-label="Toggle navigation">
<i class="fa fa-bars navbar-toggler-icon"></i>
</button>
<div class="collapse navbar-collapse" id="navbarSupportedContent">
<ul class="navbar-nav">
<li class="nav-item dropdown">
<a class="nav-link dropdown-toggle" href="#" role="button" data-bs-toggle="dropdown" aria-expanded="false">About</a>
<ul class="dropdown-menu">
<li>
<a class="dropdown-item" href="/what-is-flink/flink-architecture/">Architecture</a>
</li>
<li>
<a class="dropdown-item" href="/what-is-flink/flink-applications/">Applications</a>
</li>
<li>
<a class="dropdown-item" href="/what-is-flink/flink-operations/">Operations</a>
</li>
<li>
<a class="dropdown-item" href="/what-is-flink/use-cases/">Use Cases</a>
</li>
<li>
<a class="dropdown-item" href="/what-is-flink/powered-by/">Powered By</a>
</li>
<li>
<a class="dropdown-item" href="/what-is-flink/roadmap/">Roadmap</a>
</li>
<li>
<a class="dropdown-item" href="/what-is-flink/community/">Community & Project Info</a>
</li>
<li>
<a class="dropdown-item" href="/what-is-flink/security/">Security</a>
</li>
<li>
<a class="dropdown-item" href="/what-is-flink/special-thanks/">Special Thanks</a>
</li>
</ul>
</li>
<li class="nav-item dropdown">
<a class="nav-link dropdown-toggle" href="#" role="button" data-bs-toggle="dropdown" aria-expanded="false">Getting Started</a>
<ul class="dropdown-menu">
<li>
<a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-docs-stable/docs/try-flink/local_installation/">With Flink<i class="link fa fa-external-link title" aria-hidden="true"></i>
</a>
</li>
<li>
<a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-stable/docs/try-flink-kubernetes-operator/quick-start/">With Flink Kubernetes Operator<i class="link fa fa-external-link title" aria-hidden="true"></i>
</a>
</li>
<li>
<a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/get-started/introduction/">With Flink CDC<i class="link fa fa-external-link title" aria-hidden="true"></i>
</a>
</li>
<li>
<a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-ml-docs-stable/docs/try-flink-ml/quick-start/">With Flink ML<i class="link fa fa-external-link title" aria-hidden="true"></i>
</a>
</li>
<li>
<a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-statefun-docs-stable/getting-started/project-setup.html">With Flink Stateful Functions<i class="link fa fa-external-link title" aria-hidden="true"></i>
</a>
</li>
<li>
<a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-docs-stable/docs/learn-flink/overview/">Training Course<i class="link fa fa-external-link title" aria-hidden="true"></i>
</a>
</li>
</ul>
</li>
<li class="nav-item dropdown">
<a class="nav-link dropdown-toggle" href="#" role="button" data-bs-toggle="dropdown" aria-expanded="false">Documentation</a>
<ul class="dropdown-menu">
<li>
<a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-docs-stable/">Flink 1.19 (stable)<i class="link fa fa-external-link title" aria-hidden="true"></i>
</a>
</li>
<li>
<a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-docs-master/">Flink Master (snapshot)<i class="link fa fa-external-link title" aria-hidden="true"></i>
</a>
</li>
<li>
<a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-stable/">Kubernetes Operator 1.8 (latest)<i class="link fa fa-external-link title" aria-hidden="true"></i>
</a>
</li>
<li>
<a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main">Kubernetes Operator Main (snapshot)<i class="link fa fa-external-link title" aria-hidden="true"></i>
</a>
</li>
<li>
<a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-cdc-docs-stable">CDC 3.0 (stable)<i class="link fa fa-external-link title" aria-hidden="true"></i>
</a>
</li>
<li>
<a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-cdc-docs-master">CDC Master (snapshot)<i class="link fa fa-external-link title" aria-hidden="true"></i>
</a>
</li>
<li>
<a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-ml-docs-stable/">ML 2.3 (stable)<i class="link fa fa-external-link title" aria-hidden="true"></i>
</a>
</li>
<li>
<a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-ml-docs-master">ML Master (snapshot)<i class="link fa fa-external-link title" aria-hidden="true"></i>
</a>
</li>
<li>
<a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-statefun-docs-stable/">Stateful Functions 3.3 (stable)<i class="link fa fa-external-link title" aria-hidden="true"></i>
</a>
</li>
<li>
<a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-statefun-docs-master">Stateful Functions Master (snapshot)<i class="link fa fa-external-link title" aria-hidden="true"></i>
</a>
</li>
</ul>
</li>
<li class="nav-item dropdown">
<a class="nav-link dropdown-toggle" href="#" role="button" data-bs-toggle="dropdown" aria-expanded="false">How to Contribute</a>
<ul class="dropdown-menu">
<li>
<a class="dropdown-item" href="/how-to-contribute/overview/">Overview</a>
</li>
<li>
<a class="dropdown-item" href="/how-to-contribute/contribute-code/">Contribute Code</a>
</li>
<li>
<a class="dropdown-item" href="/how-to-contribute/reviewing-prs/">Review Pull Requests</a>
</li>
<li>
<a class="dropdown-item" href="/how-to-contribute/code-style-and-quality-preamble/">Code Style and Quality Guide</a>
</li>
<li>
<a class="dropdown-item" href="/how-to-contribute/contribute-documentation/">Contribute Documentation</a>
</li>
<li>
<a class="dropdown-item" href="/how-to-contribute/documentation-style-guide/">Documentation Style Guide</a>
</li>
<li>
<a class="dropdown-item" href="/how-to-contribute/improve-website/">Contribute to the Website</a>
</li>
<li>
<a class="dropdown-item" href="/how-to-contribute/getting-help/">Getting Help</a>
</li>
</ul>
</li>
<li class="nav-item">
<a class="nav-link" href="/posts/">Flink Blog</a>
</li>
<li class="nav-item">
<a class="nav-link" href="/downloads/">Downloads</a>
</li>
</ul>
<div class="book-search">
<div class="book-search-spinner hidden">
<i class="fa fa-refresh fa-spin"></i>
</div>
<form class="search-bar d-flex" onsubmit="return false;"su>
<input type="text" id="book-search-input" placeholder="Search" aria-label="Search" maxlength="64" data-hotkeys="s/">
<i class="fa fa-search search"></i>
<i class="fa fa-circle-o-notch fa-spin spinner"></i>
</form>
<div class="book-search-spinner hidden"></div>
<ul id="book-search-results"></ul>
</div>
</div>
</div>
</nav>
<div class="navbar-clearfix"></div>
</header>
<main class="flex">
<section class="container book-page">
<article class="markdown">
<h1>
<a href="/2022/05/05/announcing-the-release-of-apache-flink-1.15/">Announcing the Release of Apache Flink 1.15</a>
</h1>
May 5, 2022 -
Joe Moser
<a href="https://twitter.com/JoemoeAT">(@JoemoeAT)</a>
Yun Gao
<a href="https://twitter.com/YunGao16">(@YunGao16)</a>
<p><p>Thanks to our well-organized and open community, Apache Flink continues
<a href="https://www.apache.org/foundation/docs/FY2021AnnualReport.pdf">to grow</a> as a
technology and remain one of the most active projects in
the Apache community. With the release of Flink 1.15, we are proud to announce a number of
exciting changes.</p>
<p>One of the main concepts that makes Apache Flink stand out is the unification of
batch (aka bounded) and stream (aka unbounded) data processing, which helps reduce the complexity of development. A lot of
effort went into this unification in the previous releases, and you can expect more efforts in this direction.</p>
<p>Apache Flink is not only growing when it comes to contributions and users, but
also out of the original use cases. We are seeing a trend towards more business/analytics
use cases implemented in low-/no-code. Flink SQL is the feature in the Flink ecosystem
that enables such uses cases and this is why its popularity continues to grow.</p>
<p>Apache Flink is an essential building block in data pipelines/architectures and
is used with many other technologies in order to drive all sorts of use cases. While new ideas/products
may appear in this domain, existing technologies continue to establish themselves as standards for solving
mission-critical problems. Knowing that we have such a wide reach and play a role in the success of many
projects, it is important that the experience of
integrating Apache Flink with the cloud infrastructures and existing systems is as seamless and easy as possible.</p>
<p>In the 1.15 release the Apache Flink community made significant progress across all
these areas. Still those are not the only things that made it into 1.15. The
contributors improved the experience of operating Apache Flink by making it much
easier and more transparent to handle checkpoints and savepoints and their ownership,
making auto scaling more seamless and complete, by removing side effects of use cases
in which different data sources produce varying amounts of data, and - finally - the
ability to upgrade SQL jobs without losing the state. By continuing on supporting
checkpoints after tasks finished and adding window table valued functions in batch
mode, the experience of unified stream and batch processing was once more improved
making hybrid use cases way easier. In the SQL space, not only the first step in
version upgrades have been added but also JSON functions to make it easier to import
and export structured data in SQL. Both will allow users to better rely on Flink SQL
for production use cases in the long term. To establish Apache Flink as part of the
data processing ecosystem we improved the cloud interoperability and added more sink
connectors and formats. And yes we enabled a Scala-free runtime
(<a href="https://flink.apache.org/2022/02/22/scala-free.html">the hype is real</a>).</p>
<h2 id="operating-apache-flink-with-ease">
Operating Apache Flink with ease
<a class="anchor" href="#operating-apache-flink-with-ease">#</a>
</h2>
<p>Even Flink jobs that have been built and tuned by the best engineering teams still need to
be operated, usually on a long-term basis. The many deployment
patterns, APIs, tunable configs, and use cases covered by Apache Flink mean that operation
support is vital and can be burdensome.</p>
<p>In this release, we listened to user feedback and now operating Flink is made much
easier. It is now more transparent in terms of handling checkpoints and savepoints and their ownership,
which makes auto-scaling more seamless and complete (by removing side effects of use cases
where different data sources produce varying amounts of data) and enables the<br>
ability to upgrade SQL jobs without losing the state.</p>
<h3 id="clarification-of-checkpoint-and-savepoint-semantics">
Clarification of checkpoint and savepoint semantics
<a class="anchor" href="#clarification-of-checkpoint-and-savepoint-semantics">#</a>
</h3>
<p>An essential cornerstone of Flink’s fault tolerance strategy is based on
<a href="https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/ops/state/checkpoints/">checkpoints</a> and
<a href="https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/ops/state/savepoints/">savepoints</a> (see <a href="https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/ops/state/checkpoints_vs_savepoints/">the comparison</a>).
The purpose of savepoints has always been to put transitions,
backups, and upgrades of Flink jobs in the control of users. Checkpoints, on
the other hand, are intended to be fully controlled by Flink and guarantee fault
tolerance through fast recovery, failover, etc. Both concepts are quite similar, and
the underlying implementation also shares aspects of the same ideas.</p>
<p>However, both concepts grew apart by following specific feature requests and sometimes
neglecting the overarching idea and strategy. Based on user feedback, it became apparent that this should be
aligned and harmonized better and, above all, to make more clear!</p>
<p>There have been situations in which users relied on checkpoints to stop/restart jobs when savepoints would have
been the right way to go. It was also not clear that savepoints are slower since they don’t include
some of the features that make taking checkpoints so fast. In some cases like
resuming from a retained checkpoint - where the checkpoint is somehow considered as
a savepoint - it is unclear to the user when they can actually clean it up.</p>
<p>With <a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-193%3A&#43;Snapshots&#43;ownership">FLIP-193 (Snapshots ownership)</a>
the community aims to make ownership the only difference between savepoints and
checkpoints. In the 1.15 release the community has fixed some of those shortcomings
by supporting
<a href="https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/ops/state/savepoints/#savepoint-format">native and incremental savepoints</a>.
Savepoints always used to use the
canonical format which made them slower. Also writing full savepoints for sure takes
longer than doing it in an incremental way. With 1.15 if users use the native format
to take savepoints as well as the RocksDB state backend, savepoints will be
automatically taken in an incremental manner. The documentation has also been
clarified to provide a better overview and understanding of the differences between
checkpoints and savepoints. The semantics for
<a href="https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/ops/state/savepoints/#resuming-from-savepoints">resuming from savepoint/retained checkpoint</a>
have also been clarified introducing the CLAIM and NO_CLAIM mode. With
the CLAIM mode Flink takes over ownership of an existing snapshot, with NO_CLAIM it
creates its own copy and leaves the existing one up to the user. Please note that
NO_CLAIM mode is the new default behavior. The old semantic of resuming from
savepoint/retained checkpoint is still accessible but has to be manually selected by
choosing LEGACY mode.</p>
<h3 id="elastic-scaling-with-reactive-mode-and-the-adaptive-scheduler">
Elastic scaling with reactive mode and the adaptive scheduler
<a class="anchor" href="#elastic-scaling-with-reactive-mode-and-the-adaptive-scheduler">#</a>
</h3>
<p>Driven by the increasing number of cloud services built on top of Apache Flink, the
project is becoming more and more cloud native which makes elastic
scaling even more important.</p>
<p>This release improves metrics for the <a href="https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/elastic_scaling/#reactive-mode">reactive mode</a>, which is a job-scope mode where the JobManager will try to use all TaskManager resources available. To do this, we made all the metrics in
the Job scope work correctly when reactive mode is enabled.</p>
<p>We also added an exception history for the <a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-160%3A&#43;Adaptive&#43;Scheduler">adaptive scheduler</a>, which is a new scheduler that first declares the required resources and waits for them before deciding on the parallelism with which to execute a job.</p>
<p>Furthermore, downscaling is sped up significantly. The TaskManager now has a dedicated
shutdown code path, where it actively deregisters itself from the cluster instead
of relying on heartbeats, giving the JobManager a clear signal for downscaling.</p>
<h3 id="adaptive-batch-scheduler">
Adaptive batch scheduler
<a class="anchor" href="#adaptive-batch-scheduler">#</a>
</h3>
<p>In 1.15, we introduced a new scheduler to Apache Flink: the
<a href="https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/elastic_scaling/#adaptive-batch-scheduler">Adaptive Batch Scheduler</a>.
The new scheduler can automatically decide parallelisms of job vertices for batch jobs,
according to the size of data volume each vertex needs to process.</p>
<p>Major benefits of this scheduler includes:</p>
<ol>
<li>Ease-of-use: Batch job users can be relieved from parallelism tuning.</li>
<li>Adaptive: Automatically tuned parallelisms can better fit consumed datasets which
have a varying volume size every day.</li>
<li>Fine-grained: Parallelism of each job vertex will be tuned individually. This allows
vertices of SQL batch jobs to be automatically assigned different proper parallelisms.</li>
</ol>
<h3 id="watermark-alignment-across-data-sources">
Watermark alignment across data sources
<a class="anchor" href="#watermark-alignment-across-data-sources">#</a>
</h3>
<p>Having data sources that increase watermarks at different paces could lead to
problems with downstream operators. For example, some operators might need to buffer excessive
amounts of data which could lead to huge operator states. This is why we introduced watermark alignment
in this release.</p>
<p>For sources based on the new source interface,
<a href="https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/datastream/event-time/generating_watermarks/#watermark-alignment-_beta_">watermark alignment</a>
can be activated. Users can define
alignment groups to pause consuming from sources which are too far ahead from others. The ideal scenario for aligned watermarks is when there are two or more
sources that produce watermarks at a different speed and when the source has the same
parallelism as splits/shards/partitions.</p>
<h3 id="sql-version-upgrades">
SQL version upgrades
<a class="anchor" href="#sql-version-upgrades">#</a>
</h3>
<p>The execution plan of SQL queries and its resulting topology is based on optimization
rules and a cost model. This means that even minimal changes could introduce a completely
different topology. This dynamism makes guaranteeing snapshot compatibility
very challenging across different Flink versions. In the efforts of 1.15, the community focused
on keeping the same query (via the same topology) up and running even after upgrades.</p>
<p>At the core of SQL upgrades are JSON plans
(<a href="https://nightlies.apache.org/flink/flink-docs-release-1.15/api/java/org/apache/flink/table/api/CompiledPlan.html">please note that we only have documentation in our JavaDocs for now and are still working on updating the documentation</a>), which are JSON functions that make it easier to import and export structured data in SQL. This has been introduced for
internal use already in previous releases and will now be exposed externally. Both the Table API
and SQL will provide a way to compile and execute a plan which guarantees the same
topology throughout different versions. This feature will be released as an experimental MVP.
Users who want to give it a try already can create a JSON plan that can then be used
to restore a Flink job based on the old operator structure. The full feature can be expected
in Flink 1.16.</p>
<p>Reliable upgrades makes Flink SQL more dependable for production use cases in the long term.</p>
<h3 id="changelog-state-backend">
Changelog state backend
<a class="anchor" href="#changelog-state-backend">#</a>
</h3>
<p>In Flink 1.15, we introduced the MVP feature of the
<a href="https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/ops/state/state_backends/#enabling-changelog">changelog state backend</a>,
which aims at
making checkpoint intervals shorter and more predictable with the following advantages:</p>
<ol>
<li>Shorter end-to-end latency: end-to-end latency mostly depends on the checkpointing
mechanism, especially for transactional sinks. Transactional sinks commit on
checkpoints, so faster checkpoints mean more frequent commits.</li>
<li>More predictable checkpoint intervals: currently checkpointing time largely depends
on the size of the artifacts that need to be persisted on the checkpoint storage.
By keeping the size consistently small, checkpointing time becomes more predictable.</li>
<li>Less work on recovery: with more frequently checkpoints are taken, less data need
to be re-processed after each recovery.</li>
</ol>
<p>The changelog state backend helps achieve the above by continuously
persisting state changes on non-volatile storage while performing state materialization
in the background.</p>
<h3 id="repeatable-cleanup">
Repeatable cleanup
<a class="anchor" href="#repeatable-cleanup">#</a>
</h3>
<p>In previous releases of Flink, cleaning up job-related artifacts was done only once which might have resulted in abandoned artifacts in case of an error. In this version, Flink will try to run the cleanup again to avoid leaving artifacts behind. This retry mechanism runs until it was successful, by default. Users can change this behavior by configuring the <a href="https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/config/#retryable-cleanup">repeatable cleanup options</a>. Disabling the retry strategy will lead to Flink behaving like in previous releases.</p>
<p>There is still work in progress around cleaning up checkpoints, which is covered by
<a href="https://issues.apache.org/jira/browse/FLINK-26606">FLINK-26606</a>.</p>
<h3 id="openapi">
OpenAPI
<a class="anchor" href="#openapi">#</a>
</h3>
<p>Flink is now providing an experimental REST API specification following the
<a href="https://www.openapis.org">OpenAPI</a> standard.
This allows the REST API to be used with standard tools that are implementing the
OpenAPI standard.
You can find the specification <a href="https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jobmanager">here</a>.</p>
<h3 id="improvements-to-application-mode">
Improvements to application mode
<a class="anchor" href="#improvements-to-application-mode">#</a>
</h3>
<p>When running Flink in <a href="https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/overview/">application mode</a>, it can now be guaranteed that jobs will take a savepoint after they are completed if they have been configured to do so
(<a href="https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/config/#execution-shutdown-on-application-finish">see execution.shutdown-on-application-finish</a>).</p>
<p>The recovery and clean up of jobs running in application mode has also been improved. The local
state can be persisted in the working directory, which makes recovering
from local storage easier.</p>
<h2 id="unification-of-stream-and-batch-processing---more-progress">
Unification of stream and batch processing - more progress
<a class="anchor" href="#unification-of-stream-and-batch-processing---more-progress">#</a>
</h2>
<p>In the latest release, we picked up new efforts and continued some previous ones in the goal of unifying stream and batch processing.</p>
<h3 id="final-checkpoints">
Final checkpoints
<a class="anchor" href="#final-checkpoints">#</a>
</h3>
<p>In Flink 1.14, final checkpoints were added as a feature that had to be enabled manually.
Since the last release, we listened to user feedback and decided to enable it by default. For more
information and how to disable this feature, please refer to the
<a href="https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/datastream/fault-tolerance/checkpointing/#checkpointing-with-parts-of-the-graph-finished">documentation</a>.
This change in configuration can prolong the shutting down sequence of bounded
streaming jobs because jobs have to wait for a final checkpoint before being allowed to
finish.</p>
<h3 id="window-table-valued-functions">
Window table-valued functions
<a class="anchor" href="#window-table-valued-functions">#</a>
</h3>
<p><a href="https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/table/sql/queries/window-tvf/">Window table-valued functions</a>
have only been available for unbounded data streams.
With this release they will also be usable in BATCH mode. While working on this,
change window table-valued functions have also been improved in general by implementing
a dedicated operator which no longer requires those window functions to be used with
aggregators.</p>
<h2 id="flink-sql">
Flink SQL
<a class="anchor" href="#flink-sql">#</a>
</h2>
<p>Community metrics indicate that Flink SQL is widely used and becomes more popular every day. The community made several improvements but we’d like to go into two in more detail.</p>
<h3 id="casttype-system-enhancements">
CAST/Type system enhancements
<a class="anchor" href="#casttype-system-enhancements">#</a>
</h3>
<p>Data appears in all sorts and shapes but is often not in the type that you need
it to be, which is why
<a href="https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/table/types/#casting">casting</a>
is one of the most common operations in SQL. In Flink
1.15, the default behavior of a failing CAST has changed from returning a null to
returning an error, which makes it more compliant with the SQL standard. The old
casting behavior can still be used by calling the newly introduced TRY_CAST function
or restored via a configuration flag.</p>
<p>In addition, many bugs have been fixed and improvements made to the casting
functionality, to ensure correct results.</p>
<h3 id="json-functions">
JSON functions
<a class="anchor" href="#json-functions">#</a>
</h3>
<p>JSON is one of the most popular data formats and SQL users increasingly need to build
and read these data structures. Multiple
<a href="https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/table/functions/systemfunctions/#json-functions">JSON</a>
functions have been added to Flink SQL
according to the SQL 2016 standard. It allows users to inspect, create, and modify JSON
strings using the Flink SQL dialect.</p>
<h2 id="community-enablement">
Community enablement
<a class="anchor" href="#community-enablement">#</a>
</h2>
<p>Enabling people to build streaming data pipelines to solve their use cases is our goal.
The community is well aware that a
technology like Apache Flink is never used on its own and will always be part of a
bigger architecture. Thus, it is important that Flink operates well in the cloud,
connects seamlessly to other systems, and continues to support programming languages like Java and Python.</p>
<h3 id="cloud-interoperability">
Cloud interoperability
<a class="anchor" href="#cloud-interoperability">#</a>
</h3>
<p>There are users operating Flink deployments in cloud infrastructures from various
cloud providers. There are also services that offer to manage Flink deployments for
users on their platform.</p>
<p>In Flink 1.15, a recoverable writer for Google Cloud Storage has
been added. We also organized the connectors in the Flink ecosystem and put some focus
on connectors for the AWS ecosystem (i.e.
<a href="https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/datastream/kinesis/">KDS</a>,
<a href="https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/datastream/firehose/">Firehose</a>).</p>
<h3 id="the-elasticsearch-sink">
The Elasticsearch sink
<a class="anchor" href="#the-elasticsearch-sink">#</a>
</h3>
<p>There was significant work on Flink’s overall connector ecosystem, but we want to highlight the
<a href="https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/datastream/elasticsearch/">Elasticsearch sink</a> because it was implemented with
the new connector interfaces, which offers asynchronous functionality coupled with end-to-end semantics.
This sink will act as a template in the future.</p>
<h3 id="a-scala-free-flink">
A Scala-free Flink
<a class="anchor" href="#a-scala-free-flink">#</a>
</h3>
<p>A detailed
<a href="https://flink.apache.org/2022/02/22/scala-free.html">blog post</a><br>
already explains the ins and outs of why Scala users can now use the Flink
Java API with any Scala version (including Scala 3).</p>
<p>In the end, removing Scala is just part of a larger effort of cleaning up and updating
various technologies from the Flink ecosystem.</p>
<p>Starting in Flink 1.14, we removed the Mesos integration, isolated Akka, deprecated the
DataSet Java API, and hid the Table API behind an abstraction. There’s already a lot of traction in the community towards these endeavors.</p>
<h2 id="pyflink">
PyFlink
<a class="anchor" href="#pyflink">#</a>
</h2>
<p>Before Flink 1.15, Python user-defined functions were executed in separate Python
processes which caused additional serialization/deserialization and communication overhead.
In scenarios in with large amounts of data, e.g. image processing, etc, this overhead
becomes non-negligible. Besides, since it involves inter-process
communication, the processing latency is also non-negligible, which is unacceptable in
scenarios for which latency is critical, e.g. quantitative transaction, etc. In Flink
1.15, we have introduced a new execution mode named &rsquo;thread&rsquo; mode, for which Python
user-defined functions will be executed in the JVM as a thread instead of a separate
Python process. Benchmarks have shown that throughput could be increased by 2x in
common scenarios such as JSON processing. Processing latency is also decreased from
several seconds to micro-seconds. It should be noted that since this is still the first
release of &rsquo;thread&rsquo; mode, it currently only supports Python ScalarFunction which is used
in Python Table API &amp; SQL. We&rsquo;re planning to extend it to other areas in which Python
user-defined functions could be used in the next releases.</p>
<h3 id="other">
Other
<a class="anchor" href="#other">#</a>
</h3>
<p>Further work has been done on the
<a href="https://github.com/PatrickRen/flink/tree/master/flink-test-utils-parent/flink-connector-testing">connector testing framework</a>. If you want to contribute a connector or improve on one, you should definitely have a
look.</p>
<p>Some long-awaited features have been added, including the
<a href="https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/datastream/formats/csv/">CSV format</a>
and the <a href="https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/datastream/filesystem/#compaction">small file compaction</a>
in the unified sink interface.</p>
<p>The sink API has been upgraded
to <a href="https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/connector/sink2/StatefulSink.java">version 2</a>
and we encourage every connector maintainer to upgrade to this version.</p>
<h2 id="summary">
Summary
<a class="anchor" href="#summary">#</a>
</h2>
<p>Apache Flink is now easier to operate, made even more progress towards aligning stream and
batch processing, became more accessible through improvements in the SQL components,
and now integrates better with other technologies.</p>
<p>It is also worth mentioning that
the community has set up a new home for
the <a href="https://ververica.github.io/flink-cdc-connectors/release-2.1/index.html">CDC connectors</a>,
the <a href="https://cwiki.apache.org/confluence/display/FLINK/Connectors">connector repository</a>
will be externalized
(<a href="https://github.com/apache/flink-connector-elasticsearch/">with the Elasticsearch sink as a first example</a>),
and there is now a
<a href="https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/">Kubernetes operator</a>
(<a href="https://flink.apache.org/news/2022/04/03/release-kubernetes-operator-0.1.0.html">announcement blogpost</a>
maintained by the community.</p>
<p>Moving forward, the community will continue to focus on making Apache Flink a true
unified stream and batch processor and work on better integrating Flink into the cloud-native
ecosystem.</p>
<h2 id="upgrade-notes">
Upgrade Notes
<a class="anchor" href="#upgrade-notes">#</a>
</h2>
<p>While we aim to make upgrades as smooth as possible, some of the changes require users
to adjust some parts of the program when upgrading to Apache Flink 1.15. Please take a
look at the
<a href="https://nightlies.apache.org/flink/flink-docs-release-1.15/release-notes/flink-1.15/">release notes</a>
for a list of applicable adjustments and issues during
upgrades. The one big thing worth mentioning when upgrading is the updated dependencies
without the Scala version.
<a href="https://flink.apache.org/2022/02/22/scala-free.html">Get the details here.</a></p>
<h2 id="list-of-contributors">
List of Contributors
<a class="anchor" href="#list-of-contributors">#</a>
</h2>
<p>The Apache Flink community would like to thank each and every one of the contributors that have
made this release possible:</p>
<p>Ada Wong, Ahmed Hamdy, Aitozi, Alexander Fedulov, Alexander Preuß, Alexander Trushev, Ali Bahadir Zeybek,
Anton Kalashnikov, Arvid Heise, Bernard Joseph Jean Bruno, Bo Cui, Brian Zhou, Camile, ChangLi, Chengkai Yang,
Chesnay Schepler, Daisy T, Danny Cranmer, David Anderson, David Moravek, David N Perkins, Dawid Wysakowicz,
Denis-Cosmin Nutiu, Dian Fu, Dong Lin, Eelis Kostiainen, Etienne Chauchot, Fabian Paul, Francesco Guardiani,
Gabor Somogyi, Galen Warren, Gao Yun, Gen Luo, GitHub, Gyula Fora, Hang Ruan, Hangxiang Yu, Honnix, Horace Lee,
Ingo Bürk, JIN FENG, Jack, Jane Chan, Jark Wu, JianZhangYang, Jiangjie (Becket) Qin, JianzhangYang, Jiayi Liao,
Jing, Jing Ge, Jing Zhang, Jingsong Lee, JingsongLi, Jinzhong Li, Joao Boto, Joey Lee, John Karp, Jon Gillham,
Jun Qin, Junfan Zhang, Juntao Hu, Kexin, Kexin Hui, Kirill Listopad, Konstantin Knauf, LB-Yu, Leonard Xu, Lijie Wang,
Liu Jiangang, Maciej Bryński, Marios Trivyzas, MartijnVisser, Mason Chen, Matthias Pohl, Michal Ciesielczyk, Mika,
Mika Naylor, Mrart, Mulavar, Nick Burkard, Nico Kruber, Nicolas Raga, Nicolaus Weidner, Niklas Semmler, Nikolay,
Nuno Afonso, Oleg Smirnov, Paul Lin, Paul Zhang, PengFei Li, Piotr Nowojski, Px, Qingsheng Ren, Robert Metzger,
Roc Marshal, Roman, Roman Khachatryan, Ruanshubin, Rudi Kershaw, Rui Li, Ryan Scudellari, Ryan Skraba,
Sebastian Mattheis, Sergey, Sergey Nuyanzin, Shen Zhu, Shengkai, Shuo Cheng, Sike Bai, SteNicholas, Steffen Hausmann,
Stephan Ewen, Tartarus0zm, Thesharing, Thomas Weise, Till Rohrmann, Timo Walther, Tony Wei, Victor Xu,
Wenhao Ji, X-czh, Xianxun Ye, Xin Yu, Xinbin Huang, Xintong Song, Xuannan, Yang Wang, Yangze Guo, Yao Zhang,
Yi Tang, Yibo Wen, Yuan Mei, Yuanhao Tian, Yubin Li, Yuepeng Pan, Yufan Sheng, Yufei Zhang, Yuhao Bi, Yun Gao,
Yun Tang, Yuval Itzchakov, Yuxin Tan, Zakelly, Zhu Zhu, Zichen Liu, Zongwen Li, atptour2017, baisike, bgeng777,
camilesing, chenxyz707, chenzihao, chuixue, dengziming, dijkwxyz, fanrui, fengli, fenyi, fornaix, gaurav726,
godfrey he, godfreyhe, gongzhongqiang, haochenhao, hapihu, hehuiyuan, hongshuboy, huangxingbo, huweihua, iyupeng,
jiaoqingbo, jinfeng, jxjgsylsg, kevin.cyj, kylewang, lbb, liliwei, liming.1018, lincoln lee, liufangqi, liujiangang,
liushouwei, liuyongvs, lixiaobao14, lmagic233, lovewin99, lujiefsi, luoyuxia, lz, mans2singh, martijnvisser, mayue.fight,
nanmu42, oogetyboogety, paul8263, pusheng.li01, qianchutao, realdengziqi, ruanhang1993, sammieliu, shammon, shihong90,
shitou, shouweikun, shouzuo1, shuo.cs, siavash119, simenliuxing, sjwiesman, slankka, slinkydeveloper, snailHumming,
snuyanzin, sujun, sujun1, syhily, tsreaper, txdong-sz, unknown, vahmed-hamdy, wangfeifan, wangpengcheng, wangyang0918,
wangzhiwu, wangzhuo, wgzhao, wsz94, xiangqiao123, xmarker, xuyang, xuyu, xuzifu666, yangjunhan, yangze.gyz, ysymi,
yuxia Luo, zhang chaoming, zhangchaoming, zhangjiaogg, zhangjingcun, zhangjun02, zhangmang, zlzhang0122, zoucao, zp,
zzccctv, 周平, 子扬, 李锐, 蒋龙, 龙三, 庄天翼</p>
</p>
</article>
<div class="edit-this-page">
<p>
<a href="https://cwiki.apache.org/confluence/display/FLINK/Flink+Translation+Specifications">Want to contribute translation?</a>
</p>
<p>
<a href="//github.com/apache/flink-web/edit/asf-site/docs/content/posts/2022-05-05-1.15-announcement.md">
Edit This Page<i class="fa fa-edit fa-fw"></i>
</a>
</p>
</div>
</section>
<aside class="book-toc">
<nav id="TableOfContents"><h3>On This Page <a href="javascript:void(0)" class="toc" onclick="collapseToc()"><i class="fa fa-times" aria-hidden="true"></i></a></h3>
<ul>
<li>
<ul>
<li><a href="#operating-apache-flink-with-ease">Operating Apache Flink with ease</a>
<ul>
<li><a href="#clarification-of-checkpoint-and-savepoint-semantics">Clarification of checkpoint and savepoint semantics</a></li>
<li><a href="#elastic-scaling-with-reactive-mode-and-the-adaptive-scheduler">Elastic scaling with reactive mode and the adaptive scheduler</a></li>
<li><a href="#adaptive-batch-scheduler">Adaptive batch scheduler</a></li>
<li><a href="#watermark-alignment-across-data-sources">Watermark alignment across data sources</a></li>
<li><a href="#sql-version-upgrades">SQL version upgrades</a></li>
<li><a href="#changelog-state-backend">Changelog state backend</a></li>
<li><a href="#repeatable-cleanup">Repeatable cleanup</a></li>
<li><a href="#openapi">OpenAPI</a></li>
<li><a href="#improvements-to-application-mode">Improvements to application mode</a></li>
</ul>
</li>
<li><a href="#unification-of-stream-and-batch-processing---more-progress">Unification of stream and batch processing - more progress</a>
<ul>
<li><a href="#final-checkpoints">Final checkpoints</a></li>
<li><a href="#window-table-valued-functions">Window table-valued functions</a></li>
</ul>
</li>
<li><a href="#flink-sql">Flink SQL</a>
<ul>
<li><a href="#casttype-system-enhancements">CAST/Type system enhancements</a></li>
<li><a href="#json-functions">JSON functions</a></li>
</ul>
</li>
<li><a href="#community-enablement">Community enablement</a>
<ul>
<li><a href="#cloud-interoperability">Cloud interoperability</a></li>
<li><a href="#the-elasticsearch-sink">The Elasticsearch sink</a></li>
<li><a href="#a-scala-free-flink">A Scala-free Flink</a></li>
</ul>
</li>
<li><a href="#pyflink">PyFlink</a>
<ul>
<li><a href="#other">Other</a></li>
</ul>
</li>
<li><a href="#summary">Summary</a></li>
<li><a href="#upgrade-notes">Upgrade Notes</a></li>
<li><a href="#list-of-contributors">List of Contributors</a></li>
</ul>
</li>
</ul>
</nav>
</aside>
<aside class="expand-toc hidden">
<a class="toc" onclick="expandToc()" href="javascript:void(0)">
<i class="fa fa-bars" aria-hidden="true"></i>
</a>
</aside>
</main>
<footer>
<div class="separator"></div>
<div class="panels">
<div class="wrapper">
<div class="panel">
<ul>
<li>
<a href="https://flink-packages.org/">flink-packages.org</a>
</li>
<li>
<a href="https://www.apache.org/">Apache Software Foundation</a>
</li>
<li>
<a href="https://www.apache.org/licenses/">License</a>
</li>
<li>
<a href="/zh/">
<i class="fa fa-globe" aria-hidden="true"></i>&nbsp;中文版
</a>
</li>
</ul>
</div>
<div class="panel">
<ul>
<li>
<a href="/what-is-flink/security">Security</a-->
</li>
<li>
<a href="https://www.apache.org/foundation/sponsorship.html">Donate</a>
</li>
<li>
<a href="https://www.apache.org/foundation/thanks.html">Thanks</a>
</li>
</ul>
</div>
<div class="panel icons">
<div>
<a href="/posts">
<div class="icon flink-blog-icon"></div>
<span>Flink blog</span>
</a>
</div>
<div>
<a href="https://github.com/apache/flink">
<div class="icon flink-github-icon"></div>
<span>Github</span>
</a>
</div>
<div>
<a href="https://twitter.com/apacheflink">
<div class="icon flink-twitter-icon"></div>
<span>Twitter</span>
</a>
</div>
</div>
</div>
</div>
<hr/>
<div class="container disclaimer">
<p>The contents of this website are © 2024 Apache Software Foundation under the terms of the Apache License v2. Apache Flink, Flink, and the Flink logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.</p>
</div>
</footer>
</body>
</html>