| CASSANDRA-14092: MAXIMUM TTL EXPIRATION DATE |
| --------------------------------------------- |
| |
| The maximum expiration timestamp that can be represented by the storage engine is |
| 2038-01-19T03:14:06+00:00, which means that INSERTS using TTL that would expire |
| after this date are not currently supported. |
| |
| # Expiration Date Overflow Policy |
| |
| We plan to lift this limitation in newer versions, but while the fix is not available, |
| operators can decide which policy to apply when dealing with inserts with TTL exceeding |
| the maximum supported expiration date: |
| - REJECT: this is the default policy and will reject any requests with expiration |
| date timestamp after 2038-01-19T03:14:06+00:00. |
| - CAP: any insert with TTL expiring after 2038-01-19T03:14:06+00:00 will expire on |
| 2038-01-19T03:14:06+00:00 and the client will receive a warning. |
| - CAP_NOWARN: same as previous, except that the client warning will not be emitted. |
| |
| These policies may be specified via the -Dcassandra.expiration_date_overflow_policy=POLICY |
| startup option. |
| |
| # Potential data loss on earlier versions |
| |
| Prior to 3.0.16 (3.0.X) and 3.11.2 (3.11.x), there was no protection against |
| INSERTS with TTL expiring after the maximum supported date, causing the expiration |
| time field to overflow and the records to expire immediately. Expired records due |
| to overflow will not be queryable and will be permanently removed after a compaction. |
| |
| 2.1.X, 2.2.X and earlier series are not subject to this bug when assertions are enabled |
| since an AssertionError is thrown during INSERT when the expiration time field overflows |
| on these versions. When assertions are disabled then it is possible to INSERT entries |
| with overflowed local expiration time and even the earlier versions are subject to data |
| loss due to this bug. |
| |
| This issue only affected INSERTs with very large TTLs, close to the maximum allowed value |
| of 630720000 seconds (20 years), starting from 2018-01-19T03:14:06+00:00. As time progresses, |
| the maximum supported TTL will be gradually reduced as the maximum expiration date approaches. |
| For instance, a user on an affected version on 2028-01-19T03:14:06 with a TTL of 10 years |
| will be affected by this bug, so we urge users of very large TTLs to upgrade to a version |
| where this issue is addressed as soon as possible. |
| |
| # Data Recovery |
| |
| SSTables from Cassandra versions prior to 2.1.20/2.2.12/3.0.16/3.11.2 containing entries |
| with overflowed expiration time that were backed up or did not go through compaction can |
| be recovered by reinserting overflowed entries with a valid expiration time and a higher |
| timestamp, since tombstones may have been generated with the original timestamp. |
| |
| To find out if an SSTable has an entry with overflowed expiration, inspect it with the |
| sstable2json tool and look for a negative "local deletion time" field. SSTables in this |
| condition should be backed up immediately, as they are subject to data loss during |
| compaction. |
| |
| A "--reinsert-overflowed-ttl" option was added to scrub to rewrite SSTables containing |
| rows with overflowed expiration time with the maximum expiration date of |
| 2038-01-19T03:14:06+00:00 and the original timestamp + 1 (ms). Two methods are offered |
| for recovery of SSTables via scrub: |
| |
| - Offline scrub: |
| - Clone the data directory tree to another location, keeping only the folders and the |
| contents of the system tables. |
| - Clone the configuration directory to another location, setting the data_file_directories |
| property to the cloned data directory in the cloned cassandra.yaml. |
| - Copy the affected SSTables to the cloned data location of the affected table. |
| - Set the environment variable CASSANDRA_CONF=<cloned configuration directory>. |
| - Execute "sstablescrub --reinsert-overflowed-ttl <keyspace> <table>". |
| WARNING: not specifying --reinsert-overflowed-ttl is equivalent to a single-sstable |
| compaction, so the data with overflowed will be removed - make sure to back up |
| your SSTables before running scrub. |
| - Once the scrub is completed, copy the resulting SSTables to the original data directory. |
| - Execute "nodetool refresh" in a live node to load the recovered SSTables. |
| |
| - Online scrub: |
| - Disable compaction on the node with "nodetool disableautocompaction" - this step is crucial |
| as otherwise, the data may be removed permanently during compaction. |
| - Copy the SSTables containing entries with overflowed expiration time to the data directory. |
| - run "nodetool refresh" to load the SSTables. |
| - run "nodetool scrub --reinsert-overflowed-ttl <keyspace> <table>". |
| - Re-enable compactions after verifying that scrub recovered the missing entries. |
| |
| See https://issues.apache.org/jira/browse/CASSANDRA-14092 for more details about this issue. |