import Tabs from ‘@theme/Tabs’; import TabItem from ‘@theme/TabItem’;
This topic describes the API endpoints for managing retention rules in Apache Druid. You can configure retention rules in the Druid web console or API.
Druid uses retention rules to determine what data is retained in the cluster. Druid supports load, drop, and broadcast rules. For more information, see Using rules to drop and retain data.
In this topic, http://ROUTER_IP:ROUTER_PORT
is a placeholder for your Router service address and port. Replace it with the information for your deployment. For example, use http://localhost:8888
for quickstart deployments.
Updates one or more retention rules for a datasource. The request body takes an array of retention rule objects. For details on defining retention rules, see the following sources:
This request overwrites any existing rules for the datasource. Druid reads rules in the order in which they appear; for more information, see rule structure.
Note that this endpoint returns an HTTP 200 OK
even if the datasource does not exist.
POST /druid/coordinator/v1/rules/:dataSource
The endpoint supports a set of optional header parameters to populate the author
and comment
fields in the auditInfo
property for audit history.
X-Druid-Author
(optional)X-Druid-Comment
(optional)Successfully updated retention rules for specified datasource
The following example sets a set of broadcast, load, and drop retention rules for the kttm1
datasource.
curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/rules/kttm1" \ --header 'X-Druid-Author: doc intern' \ --header 'X-Druid-Comment: submitted via api' \ --header 'Content-Type: application/json' \ --data '[ { "type": "broadcastForever" }, { "type": "loadForever", "tieredReplicants": { "_default_tier": 2 }, "useDefaultTierForNull": true }, { "type": "dropByPeriod", "period": "P1M" } ]'
POST /druid/coordinator/v1/rules/kttm1 HTTP/1.1 Host: http://ROUTER_IP:ROUTER_PORT X-Druid-Author: doc intern X-Druid-Comment: submitted via api Content-Type: application/json Content-Length: 273 [ { "type": "broadcastForever" }, { "type": "loadForever", "tieredReplicants": { "_default_tier": 1 }, "useDefaultTierForNull": true }, { "type": "dropByPeriod", "period": "P1M" } ]
A successful request returns an HTTP 200 OK
message code and an empty response body.
Updates one or more default retention rules for all datasources. Submit retention rules as an array of objects in the request body. For details on defining retention rules, see the following sources:
This request overwrites any existing rules for all datasources. To remove default retention rules for all datasources, submit an empty rule array in the request body. Rules are read in the order in which they appear; for more information, see rule structure.
POST /druid/coordinator/v1/rules/_default
The endpoint supports a set of optional header parameters to populate the author
and comment
fields in the auditInfo
property for audit history.
X-Druid-Author
(optional)X-Druid-Comment
(optional)Successfully updated default retention rules
Error with request body
The following example updates the default retention rule for all datasources with a loadByInterval
rule.
curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/rules/_default" \ --header 'Content-Type: application/json' \ --data '[ { "type": "loadByInterval", "tieredReplicants": {}, "useDefaultTierForNull": false, "interval": "2010-01-01/2020-01-01" } ]'
POST /druid/coordinator/v1/rules/_default HTTP/1.1 Host: http://ROUTER_IP:ROUTER_PORT Content-Type: application/json Content-Length: 205 [ { "type": "loadByInterval", "tieredReplicants": {}, "useDefaultTierForNull": false, "interval": "2010-01-01/2020-01-01" } ]
A successful request returns an HTTP 200 OK
message code and an empty response body.
Retrieves all current retention rules in the cluster including the default retention rule. Returns an array of objects for each datasource and their associated retention rules.
GET /druid/coordinator/v1/rules
Successfully retrieved retention rules
curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/rules"
GET /druid/coordinator/v1/rules HTTP/1.1 Host: http://ROUTER_IP:ROUTER_PORT
{ "_default": [ { "tieredReplicants": { "_default_tier": 2 }, "type": "loadForever" } ], "social_media": [ { "interval": "2023-01-01T00:00:00.000Z/2023-02-01T00:00:00.000Z", "type": "dropByInterval" } ], "wikipedia_api": [], }
Retrieves an array of rule objects for a single datasource. Returns an empty array if there are no retention rules.
Note that this endpoint returns an HTTP 200 OK
message code even if the datasource does not exist.
GET /druid/coordinator/v1/rules/:dataSource
full
(optional)Successfully retrieved retention rules
The following example retrieves the custom retention rules and default retention rules for datasource with the name social_media
.
curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/rules/social_media?full=null"
GET /druid/coordinator/v1/rules/social_media?full=null HTTP/1.1 Host: http://ROUTER_IP:ROUTER_PORT
[ { "interval": "2020-01-01T00:00:00.000Z/2022-02-01T00:00:00.000Z", "type": "dropByInterval" }, { "interval": "2010-01-01T00:00:00.000Z/2020-01-01T00:00:00.000Z", "tieredReplicants": { "_default_tier": 2 }, "type": "loadByInterval" }, { "tieredReplicants": { "_default_tier": 2 }, "type": "loadForever" } ]
Retrieves the audit history of rules for all datasources over an interval of time. The default interval is 1 week. You can change this period by setting druid.audit.manager.auditHistoryMillis
in the runtime.properties
file for the Coordinator.
GET /druid/coordinator/v1/rules/history
Note that the following query parameters cannot be chained.
interval
(optional)/
. For example, 2023-07-13/2023-07-19
.count
(optional)n
entries.Successfully retrieved audit history
Request in the incorrect format
count
query parameter too large
The following example retrieves the audit history for all datasources from 2023-07-13
to 2023-07-19
.
curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/rules/history?interval=2023-07-13%2F2023-07-19"
GET /druid/coordinator/v1/rules/history?interval=2023-07-13/2023-07-19 HTTP/1.1 Host: http://ROUTER_IP:ROUTER_PORT
[ { "key": "social_media", "type": "rules", "auditInfo": { "author": "console", "comment": "test", "ip": "127.0.0.1" }, "payload": "[{\"interval\":\"2023-01-01T00:00:00.000Z/2023-02-01T00:00:00.000Z\",\"type\":\"dropByInterval\"}]", "auditTime": "2023-07-13T18:05:33.066Z" }, { "key": "social_media", "type": "rules", "auditInfo": { "author": "console", "comment": "test", "ip": "127.0.0.1" }, "payload": "[]", "auditTime": "2023-07-18T18:10:21.203Z" }, { "key": "wikipedia_api", "type": "rules", "auditInfo": { "author": "console", "comment": "test", "ip": "127.0.0.1" }, "payload": "[{\"tieredReplicants\":{\"_default_tier\":2},\"type\":\"loadForever\"}]", "auditTime": "2023-07-18T18:10:44.519Z" }, { "key": "wikipedia_api", "type": "rules", "auditInfo": { "author": "console", "comment": "test", "ip": "127.0.0.1" }, "payload": "[]", "auditTime": "2023-07-18T18:11:02.110Z" }, { "key": "social_media", "type": "rules", "auditInfo": { "author": "console", "comment": "test", "ip": "127.0.0.1" }, "payload": "[{\"interval\":\"2023-07-03T18:49:54.848Z/2023-07-03T18:49:55.861Z\",\"type\":\"dropByInterval\"}]", "auditTime": "2023-07-18T18:32:50.060Z" }, { "key": "social_media", "type": "rules", "auditInfo": { "author": "console", "comment": "test", "ip": "127.0.0.1" }, "payload": "[{\"interval\":\"2020-01-01T00:00:00.000Z/2022-02-01T00:00:00.000Z\",\"type\":\"dropByInterval\"}]", "auditTime": "2023-07-18T18:34:09.657Z" }, { "key": "social_media", "type": "rules", "auditInfo": { "author": "console", "comment": "test", "ip": "127.0.0.1" }, "payload": "[{\"interval\":\"2020-01-01T00:00:00.000Z/2022-02-01T00:00:00.000Z\",\"type\":\"dropByInterval\"},{\"tieredReplicants\":{\"_default_tier\":2},\"type\":\"loadForever\"}]", "auditTime": "2023-07-18T18:38:37.223Z" }, { "key": "social_media", "type": "rules", "auditInfo": { "author": "console", "comment": "test", "ip": "127.0.0.1" }, "payload": "[{\"interval\":\"2020-01-01T00:00:00.000Z/2022-02-01T00:00:00.000Z\",\"type\":\"dropByInterval\"},{\"interval\":\"2010-01-01T00:00:00.000Z/2020-01-01T00:00:00.000Z\",\"tieredReplicants\":{\"_default_tier\":2},\"type\":\"loadByInterval\"}]", "auditTime": "2023-07-18T18:49:43.964Z" } ]