| <!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>GrokReader</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">GrokReader</h1><h2>Description: </h2><p>Provides a mechanism for reading unstructured text data, such as log files, and structuring the data so that it can be processed. The service is configured using Grok patterns. The service reads from a stream of data and splits each message that it finds into a separate Record, each containing the fields that are configured. If a line in the input does not match the expected message pattern, the line of text is either considered to be part of the previous message or is skipped, depending on the configuration, with the exception of stack traces. A stack trace that is found at the end of a log message is considered to be part of the previous message but is added to the 'stackTrace' field of the Record. If a record has no stack trace, it will have a NULL value for the stackTrace field (assuming that the schema does in fact include a stackTrace field of type String). Assuming that the schema includes a '_raw' field of type String, the raw message will be included in the Record.</p><p><a href="additionalDetails.html">Additional Details...</a></p><h3>Tags: </h3><p>grok, logs, logfiles, parse, unstructured, text, record, reader, regex, pattern, logstash</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the <a href="../../../../../html/expression-language-guide.html">NiFi Expression Language</a>.</p><table id="properties"><tr><th>Display Name</th><th>API Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name"><strong>Schema Access Strategy</strong></td><td>schema-access-strategy</td><td id="default-value">Use String Fields From Grok Expression</td><td id="allowable-values"><ul><li>Use String Fields From Grok Expression <img src="../../../../../html/images/iconInfo.png" alt="The schema will be derived using the field names present in all configured Grok Expressions. All schema fields will have a String type and will be marked as nullable. The schema will also include a `stackTrace` field, and a `_raw` field containing the input line string." title="The schema will be derived using the field names present in all configured Grok Expressions. All schema fields will have a String type and will be marked as nullable. The schema will also include a `stackTrace` field, and a `_raw` field containing the input line string."></img></li><li>Use 'Schema Name' Property <img src="../../../../../html/images/iconInfo.png" alt="The name of the Schema to use is specified by the 'Schema Name' Property. The value of this property is used to lookup the Schema in the configured Schema Registry service." title="The name of the Schema to use is specified by the 'Schema Name' Property. The value of this property is used to lookup the Schema in the configured Schema Registry service."></img></li><li>Use 'Schema Text' Property <img src="../../../../../html/images/iconInfo.png" alt="The text of the Schema itself is specified by the 'Schema Text' Property. The value of this property must be a valid Avro Schema. If Expression Language is used, the value of the 'Schema Text' property must be valid after substituting the expressions." title="The text of the Schema itself is specified by the 'Schema Text' Property. The value of this property must be a valid Avro Schema. If Expression Language is used, the value of the 'Schema Text' property must be valid after substituting the expressions."></img></li><li>HWX Schema Reference Attributes <img src="../../../../../html/images/iconInfo.png" alt="The FlowFile contains 3 Attributes that will be used to lookup a Schema from the configured Schema Registry: 'schema.identifier', 'schema.version', and 'schema.protocol.version'" title="The FlowFile contains 3 Attributes that will be used to lookup a Schema from the configured Schema Registry: 'schema.identifier', 'schema.version', and 'schema.protocol.version'"></img></li><li>HWX Content-Encoded Schema Reference <img src="../../../../../html/images/iconInfo.png" alt="The content of the FlowFile contains a reference to a schema in the Schema Registry service. The reference is encoded as a single byte indicating the 'protocol version', followed by 8 bytes indicating the schema identifier, and finally 4 bytes indicating the schema version, as per the Hortonworks Schema Registry serializers and deserializers, found at https://github.com/hortonworks/registry" title="The content of the FlowFile contains a reference to a schema in the Schema Registry service. The reference is encoded as a single byte indicating the 'protocol version', followed by 8 bytes indicating the schema identifier, and finally 4 bytes indicating the schema version, as per the Hortonworks Schema Registry serializers and deserializers, found at https://github.com/hortonworks/registry"></img></li><li>Confluent Content-Encoded Schema Reference <img src="../../../../../html/images/iconInfo.png" alt="The content of the FlowFile contains a reference to a schema in the Schema Registry service. The reference is encoded as a single 'Magic Byte' followed by 4 bytes representing the identifier of the schema, as outlined at http://docs.confluent.io/current/schema-registry/docs/serializer-formatter.html. This is based on version 3.2.x of the Confluent Schema Registry." title="The content of the FlowFile contains a reference to a schema in the Schema Registry service. The reference is encoded as a single 'Magic Byte' followed by 4 bytes representing the identifier of the schema, as outlined at http://docs.confluent.io/current/schema-registry/docs/serializer-formatter.html. This is based on version 3.2.x of the Confluent Schema Registry."></img></li></ul></td><td id="description">Specifies how to obtain the schema that is to be used for interpreting the data.</td></tr><tr><td id="name">Schema Registry</td><td>schema-registry</td><td></td><td id="allowable-values"><strong>Controller Service API: </strong><br/>SchemaRegistry<br/><strong>Implementations: </strong><a href="../../../nifi-confluent-platform-nar/1.19.1/org.apache.nifi.confluent.schemaregistry.ConfluentSchemaRegistry/index.html">ConfluentSchemaRegistry</a><br/><a href="../../../nifi-registry-nar/1.19.1/org.apache.nifi.schemaregistry.services.AvroSchemaRegistry/index.html">AvroSchemaRegistry</a><br/><a href="../../../nifi-hwx-schema-registry-nar/1.19.1/org.apache.nifi.schemaregistry.hortonworks.HortonworksSchemaRegistry/index.html">HortonworksSchemaRegistry</a></td><td id="description">Specifies the Controller Service to use for the Schema Registry<br/><br/><strong>This Property is only considered if </strong><strong>the [Schema Access Strategy] Property is set to one of the following values: [Confluent Content-Encoded Schema Reference], [Use 'Schema Name' Property], [HWX Schema Reference Attributes], [HWX Content-Encoded Schema Reference]</strong></td></tr><tr><td id="name">Schema Name</td><td>schema-name</td><td id="default-value">${schema.name}</td><td id="allowable-values"></td><td id="description">Specifies the name of the schema to lookup in the Schema Registry property<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong><br/><br/><strong>This Property is only considered if </strong><strong>the [Schema Access Strategy] Property has a value of "Use 'Schema Name' Property".</strong></td></tr><tr><td id="name">Schema Version</td><td>schema-version</td><td></td><td id="allowable-values"></td><td id="description">Specifies the version of the schema to lookup in the Schema Registry. If not specified then the latest version of the schema will be retrieved.<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong><br/><br/><strong>This Property is only considered if </strong><strong>the [Schema Access Strategy] Property has a value of "Use 'Schema Name' Property".</strong></td></tr><tr><td id="name">Schema Branch</td><td>schema-branch</td><td></td><td id="allowable-values"></td><td id="description">Specifies the name of the branch to use when looking up the schema in the Schema Registry property. If the chosen Schema Registry does not support branching, this value will be ignored.<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong><br/><br/><strong>This Property is only considered if </strong><strong>the [Schema Access Strategy] Property has a value of "Use 'Schema Name' Property".</strong></td></tr><tr><td id="name">Schema Text</td><td>schema-text</td><td id="default-value">${avro.schema}</td><td id="allowable-values"></td><td id="description">The text of an Avro-formatted Schema<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong><br/><br/><strong>This Property is only considered if </strong><strong>the [Schema Access Strategy] Property has a value of "Use 'Schema Text' Property".</strong></td></tr><tr><td id="name">Grok Patterns</td><td>Grok Pattern File</td><td></td><td id="allowable-values"></td><td id="description">Grok Patterns to use for parsing logs. If not specified, a built-in default Pattern file will be used. If specified, all patterns specified will override the default patterns. See the Controller Service's Additional Details for a list of pre-defined patterns.<br/><br/><strong>This property requires exactly one resource to be provided. That resource may be any of the following types: text, file, URL.</strong><br/><br/><strong>Supports Expression Language: true (will be evaluated using variable registry only)</strong></td></tr><tr><td id="name"><strong>Grok Expressions</strong></td><td>Grok Expression</td><td></td><td id="allowable-values"></td><td id="description">Specifies the format of a log line in Grok format. This allows the Record Reader to understand how to parse each log line. The property supports one or more Grok expressions. The Reader attempts to parse input lines according to the configured order of the expressions.If a line in the log file does not match any expressions, the line will be assumed to belong to the previous log message.If other Grok patterns are referenced by this expression, they need to be supplied in the Grok Pattern File property.<br/><br/><strong>This property requires exactly one resource to be provided. That resource may be any of the following types: text, file, URL.</strong><br/></td></tr><tr><td id="name"><strong>No Match Behavior</strong></td><td>no-match-behavior</td><td id="default-value">Append to Previous Message</td><td id="allowable-values"><ul><li>Append to Previous Message <img src="../../../../../html/images/iconInfo.png" alt="The line of text that does not match the Grok Expression will be appended to the last field of the prior message." title="The line of text that does not match the Grok Expression will be appended to the last field of the prior message."></img></li><li>Skip Line <img src="../../../../../html/images/iconInfo.png" alt="The line of text that does not match the Grok Expression will be skipped." title="The line of text that does not match the Grok Expression will be skipped."></img></li><li>Raw Line <img src="../../../../../html/images/iconInfo.png" alt="The line of text that does not match the Grok Expression will only be added to the _raw field." title="The line of text that does not match the Grok Expression will only be added to the _raw field."></img></li></ul></td><td id="description">If a line of text is encountered and it does not match the given Grok Expression, and it is not part of a stack trace, this property specifies how the text should be processed.</td></tr></table><h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.<h3>System Resource Considerations:</h3>None specified.</body></html> |