layout: post title: “Stateful Functions 2.1.0 Release Announcement” date: 2020-06-09T08:00:00.000Z categories: news authors:
The Apache Flink community is happy to announce the release of Stateful Functions (StateFun) 2.1.0! This release introduces new features around state expiration and performance improvements for co-located deployments, as well as other important changes that improve the stability and testability of the project. As the community around StateFun grows, the release cycle will follow this pattern of smaller and more frequent releases to incorporate user feedback and allow for faster iteration.
The binary distribution and source artifacts are now available on the updated Downloads page of the Flink website, and the most recent Python SDK distribution is available on PyPI. For more details, check the complete release changelog and the updated documentation. We encourage you to download the release and share your feedback with the community through the Flink mailing lists or JIRA!
{% toc %}
Being able to define state expiration and a state cleanup strategy is a useful feature for stateful applications — for example, to keep state size from growing indefinitely or to work with sensitive data. In previous StateFun versions, users could implement this behavior manually using delayed messages as state expiration callbacks. For StateFun 2.1, the community has worked on enabling users to configure any persisted state to expire and be purged after a given duration (i.e. the state time-to-live) (FLINK-17644, FLINK-17875).
Persisted state can be configured to expire after the last write operation (AFTER_WRITE) or after the last read or write operation (AFTER_READ_AND_WRITE). For the Java SDK, users can configure State TTL in the definition of their persisted fields:
@Persisted PersistedValue<Integer> table = PersistedValue.of( "my-value", Integer.class, Expiration.expireAfterWriting(Duration.ofHours(1)));
For remote functions using e.g. the Python SDK, users can configure State TTL in their module.yaml:
functions:
- function:
states:
- name: xxxx
expireAfter: 5min # optional key
Stateful functions can be deployed in multiple ways, even within the same application. For deployments where functions are co-located with the Flink StateFun workers, it’s common to use Kubernetes to deploy pods consisting of a Flink StateFun container and the function sidecar container, communicating via the pod-local network. To improve the performance of such deployments, StateFun 2.1 allows using Unix Domain Sockets (UDS) to communicate between containers in the same pod (i.e. the same machine) (FLINK-17611), which drastically reduces the overhead of going through the network stack.
Users can enable transport via UDS in a remote module by specifying the following in their module.yaml:
functions:
- function:
spec:
- endpoint: http(s)+unix://<socket-file-path>/<serve-url-path>
[FLINK-17712] The Flink version in StateFun 2.1 has been upgraded to 1.10.1, the most recent patch version.
[FLINK-17533] StateFun 2.1 now supports concurrent checkpoints, which means applications will no longer fail on savepoints that are triggered concurrently to a checkpoint.
[FLINK-16928] StateFun 2.0 was using the Flink legacy scheduler due to a bug in Flink 1.10. In 2.1, this change is reverted to using the new Flink scheduler again.
[FLINK-17516] The coverage for end-to-end StateFun tests has been extended to also include exactly-once semantics verification (with failure recovery).
Please review the release notes for a detailed list of changes and new features if you plan to upgrade your setup to Stateful Functions 2.1.
The Apache Flink community would like to thank all contributors that have made this release possible:
abc863377, Authuir, Chesnay Schepler, Congxian Qiu, David Anderson, Dian Fu, Francesco Guardiani, Igal Shilman, Marta Paes Moreira, Patrick Wiener, Rafi Aroch, Seth Wiesman, Stephan Ewen, Tzu-Li (Gordon) Tai
If you’d like to get involved, we’re always looking for new contributors — especially around SDKs for other languages like Go, Rust or Javascript.