Field | Type | Description | Required |
---|---|---|---|
useFieldDiscovery | Boolean | If true, interpret all fields with singular values (not a map or list) and flat lists (lists of singular values) at the root level as columns. | no (default == true) |
fields | JSON Object array | Specifies the fields of interest and how they are accessed | no (default == []) |
Defining the JSON Flatten Spec allows nested JSON fields to be flattened during ingestion time. Only parseSpecs of types “json” or “avro” support flattening.
‘fields’ is a list of JSON Objects, describing the field names and how the fields are accessed:
Field | Type | Description | Required |
---|---|---|---|
type | String | Type of the field, “root”, “path” or “jq”. | yes |
name | String | This string will be used as the column name when the data has been ingested. | yes |
expr | String | Defines an expression for accessing the field within the JSON object, using JsonPath notation for type “path”, and jackson-jq for type “jq”. This field is only used for type “path” and “jq”, otherwise ignored. | only for type “path” or “jq” |
Suppose the event JSON has the following form:
{ "timestamp": "2015-09-12T12:10:53.155Z", "dim1": "qwerty", "dim2": "asdf", "dim3": "zxcv", "ignore_me": "ignore this", "metrica": 9999, "foo": {"bar": "abc"}, "foo.bar": "def", "nestmet": {"val": 42}, "hello": [1.0, 2.0, 3.0, 4.0, 5.0], "mixarray": [1.0, 2.0, 3.0, 4.0, {"last": 5}], "world": [{"hey": "there"}, {"tree": "apple"}], "thing": {"food": ["sandwich", "pizza"]} }
The column “metrica” is a Long metric column, “hello” is an array of Double metrics, and “nestmet.val” is a nested Long metric. All other columns are dimensions.
To flatten this JSON, the parseSpec could be defined as follows:
"parseSpec": { "format": "json", "flattenSpec": { "useFieldDiscovery": true, "fields": [ { "type": "root", "name": "dim1" }, "dim2", { "type": "path", "name": "foo.bar", "expr": "$.foo.bar" }, { "type": "root", "name": "foo.bar" }, { "type": "path", "name": "path-metric", "expr": "$.nestmet.val" }, { "type": "path", "name": "hello-0", "expr": "$.hello[0]" }, { "type": "path", "name": "hello-4", "expr": "$.hello[4]" }, { "type": "path", "name": "world-hey", "expr": "$.world[0].hey" }, { "type": "path", "name": "worldtree", "expr": "$.world[1].tree" }, { "type": "path", "name": "first-food", "expr": "$.thing.food[0]" }, { "type": "path", "name": "second-food", "expr": "$.thing.food[1]" }, { "type": "jq", "name": "first-food-by-jq", "expr": ".thing.food[1]" }, { "type": "jq", "name": "hello-total", "expr": ".hello | sum" } ] }, "dimensionsSpec" : { "dimensions" : [], "dimensionsExclusions": ["ignore_me"] }, "timestampSpec" : { "format" : "auto", "column" : "timestamp" } }
Fields “dim3”, “ignore_me”, and “metrica” will be automatically discovered because ‘useFieldDiscovery’ is true, so they have been omitted from the field spec list.
“ignore_me” will be automatically discovered but excluded as specified by dimensionsExclusions.
Aggregators should use the metric column names as defined in the flattenSpec. Using the example above:
"metricsSpec" : [ { "type" : "longSum", "name" : "path-metric-sum", "fieldName" : "path-metric" }, { "type" : "doubleSum", "name" : "hello-0-sum", "fieldName" : "hello-0" }, { "type" : "longSum", "name" : "metrica-sum", "fieldName" : "metrica" } ]
Note that: