[STREAMSTORAGE] support an optional time to live (TTL) on a per table basis


Bookkeeper's state store is often used, especially through Pulsar
functions, to store processing state in a streaming application which
require persistent short term storage.  By allowing clients to set a
TTL, this data will get cleaned up by the system automatically.


Specify the TTL via stream.StreamConfiguration.ttlSeconds, default is an
infinite TTL.

Note that changing the TTL on existing stream will have no effect until
the stream is next loaded by a storage container.

Update RocksDB to to address failures when using TtlDb.

Descriptions of the changes in this PR:

### Motivation

(Explain: why you're making that change, what is the problem you're trying to solve)

### Changes

(Describe: what changes you have made)

Master Issue: #<master-issue-number>

Reviewers: Henry Saputra <hsaputra@apache.org>, Enrico Olivelli <eolivelli@gmail.com>, Prashant Kumar <None>

This closes #2775 from mauricebarnum/stream-table-expiration and squashes the following commits:

50471add7 [Maurice Barnum] update LICENSE files for rocksdb-
caa8532d4 [Maurice Barnum] [STATESTORE] support an optional time to live (TTL) on a per table basis
21 files changed
tree: 2440b8b8ba7c9b0bc9608012e8c28c68678e3e5b
  1. .github/
  2. .gitignore
  3. .test-infra/
  4. Jenkinsfile
  7. README.md
  8. bin/
  9. bookkeeper-benchmark/
  10. bookkeeper-common-allocator/
  11. bookkeeper-common/
  12. bookkeeper-dist/
  13. bookkeeper-http/
  14. bookkeeper-proto/
  15. bookkeeper-server/
  16. bookkeeper-stats-providers/
  17. bookkeeper-stats/
  18. build.gradle
  19. buildtools/
  20. circe-checksum/
  21. conf/
  22. cpu-affinity/
  23. dependencies.gradle
  24. deploy/
  25. dev/
  26. docker/
  27. gradle.properties
  28. gradle/
  29. gradlew
  30. gradlew.bat
  31. metadata-drivers/
  32. microbenchmarks/
  33. pom.xml
  34. settings.gradle
  35. shaded/
  36. site/
  37. site2/
  38. stats/
  39. stream/
  40. tests/
  41. tools/
  42. version.gradle

Build Status Build Status Coverage Status Maven Central

Apache BookKeeper

Apache BookKeeper is a scalable, fault tolerant and low latency storage service optimized for append-only workloads.

It is suitable for being used in following scenarios:

  • WAL (Write-Ahead-Logging), e.g. HDFS NameNode.
  • Message Store, e.g. Apache Pulsar.
  • Offset/Cursor Store, e.g. Apache Pulsar.
  • Object/Blob Store, e.g. storing state machine snapshots.

Get Started

  • Checkout the project website.
  • Concepts: Start with the basic concepts of Apache BookKeeper. This will help you to fully understand the other parts of the documentation.
  • Follow the Install guide to setup BookKeeper.


Please visit the Documentation from the project website for more information.

Get In Touch

Report a Bug

For filing bugs, suggesting improvements, or requesting new features, help us out by opening a Github issue or opening an Apache jira.

Need Help?

Subscribe or mail the user@bookkeeper.apache.org list - Ask questions, find answers, and also help other users.

Subscribe or mail the dev@bookkeeper.apache.org list - Join development discussions, propose new ideas and connect with contributors.

Join us on Slack - This is the most immediate way to connect with Apache BookKeeper committers and contributors.


We feel that a welcoming open community is important and welcome contributions.

Contributing Code

  1. See Developer Setup to get your local environment setup.

  2. Take a look at our open issues: JIRA Issues Github Issues.

  3. Review our coding style and follow our pull requests to learn about our conventions.

  4. Make your changes according to our contribution guide.

Improving Website and Documentation

  1. See Building the website and documentation on how to build the website and documentation.