| <!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>CSVReader</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">CSVReader</h1><h2>Description: </h2><p>Parses CSV-formatted data, returning each row in the CSV file as a separate record. This reader allows for inferring a schema based on the first line of the CSV, if a 'header line' is present, or providing an explicit schema for interpreting the values. See Controller Service's Usage for further documentation.</p><p><a href="additionalDetails.html">Additional Details...</a></p><h3>Tags: </h3><p>csv, parse, record, row, reader, delimited, comma, separated, values</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the <a href="../../../../../html/expression-language-guide.html">NiFi Expression Language</a>.</p><table id="properties"><tr><th>Display Name</th><th>API Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name"><strong>Schema Access Strategy</strong></td><td>schema-access-strategy</td><td id="default-value">Infer Schema</td><td id="allowable-values"><ul><li>Use 'Schema Name' Property <img src="../../../../../html/images/iconInfo.png" alt="The name of the Schema to use is specified by the 'Schema Name' Property. The value of this property is used to lookup the Schema in the configured Schema Registry service." title="The name of the Schema to use is specified by the 'Schema Name' Property. The value of this property is used to lookup the Schema in the configured Schema Registry service."></img></li><li>Use 'Schema Text' Property <img src="../../../../../html/images/iconInfo.png" alt="The text of the Schema itself is specified by the 'Schema Text' Property. The value of this property must be a valid Avro Schema. If Expression Language is used, the value of the 'Schema Text' property must be valid after substituting the expressions." title="The text of the Schema itself is specified by the 'Schema Text' Property. The value of this property must be a valid Avro Schema. If Expression Language is used, the value of the 'Schema Text' property must be valid after substituting the expressions."></img></li><li>HWX Schema Reference Attributes <img src="../../../../../html/images/iconInfo.png" alt="The FlowFile contains 3 Attributes that will be used to lookup a Schema from the configured Schema Registry: 'schema.identifier', 'schema.version', and 'schema.protocol.version'" title="The FlowFile contains 3 Attributes that will be used to lookup a Schema from the configured Schema Registry: 'schema.identifier', 'schema.version', and 'schema.protocol.version'"></img></li><li>HWX Content-Encoded Schema Reference <img src="../../../../../html/images/iconInfo.png" alt="The content of the FlowFile contains a reference to a schema in the Schema Registry service. The reference is encoded as a single byte indicating the 'protocol version', followed by 8 bytes indicating the schema identifier, and finally 4 bytes indicating the schema version, as per the Hortonworks Schema Registry serializers and deserializers, found at https://github.com/hortonworks/registry" title="The content of the FlowFile contains a reference to a schema in the Schema Registry service. The reference is encoded as a single byte indicating the 'protocol version', followed by 8 bytes indicating the schema identifier, and finally 4 bytes indicating the schema version, as per the Hortonworks Schema Registry serializers and deserializers, found at https://github.com/hortonworks/registry"></img></li><li>Confluent Content-Encoded Schema Reference <img src="../../../../../html/images/iconInfo.png" alt="The content of the FlowFile contains a reference to a schema in the Schema Registry service. The reference is encoded as a single 'Magic Byte' followed by 4 bytes representing the identifier of the schema, as outlined at http://docs.confluent.io/current/schema-registry/docs/serializer-formatter.html. This is based on version 3.2.x of the Confluent Schema Registry." title="The content of the FlowFile contains a reference to a schema in the Schema Registry service. The reference is encoded as a single 'Magic Byte' followed by 4 bytes representing the identifier of the schema, as outlined at http://docs.confluent.io/current/schema-registry/docs/serializer-formatter.html. This is based on version 3.2.x of the Confluent Schema Registry."></img></li><li>Use String Fields From Header <img src="../../../../../html/images/iconInfo.png" alt="The first non-comment line of the CSV file is a header line that contains the names of the columns. The schema will be derived by using the column names in the header and assuming that all columns are of type String." title="The first non-comment line of the CSV file is a header line that contains the names of the columns. The schema will be derived by using the column names in the header and assuming that all columns are of type String."></img></li><li>Infer Schema <img src="../../../../../html/images/iconInfo.png" alt="The Schema of the data will be inferred automatically when the data is read. See component Usage and Additional Details for information about how the schema is inferred." title="The Schema of the data will be inferred automatically when the data is read. See component Usage and Additional Details for information about how the schema is inferred."></img></li></ul></td><td id="description">Specifies how to obtain the schema that is to be used for interpreting the data.</td></tr><tr><td id="name">Schema Registry</td><td>schema-registry</td><td></td><td id="allowable-values"><strong>Controller Service API: </strong><br/>SchemaRegistry<br/><strong>Implementations: </strong><a href="../../../nifi-registry-nar/1.19.0/org.apache.nifi.schemaregistry.services.AvroSchemaRegistry/index.html">AvroSchemaRegistry</a><br/><a href="../../../nifi-confluent-platform-nar/1.19.0/org.apache.nifi.confluent.schemaregistry.ConfluentSchemaRegistry/index.html">ConfluentSchemaRegistry</a><br/><a href="../../../nifi-hwx-schema-registry-nar/1.19.0/org.apache.nifi.schemaregistry.hortonworks.HortonworksSchemaRegistry/index.html">HortonworksSchemaRegistry</a></td><td id="description">Specifies the Controller Service to use for the Schema Registry<br/><br/><strong>This Property is only considered if </strong><strong>the [Schema Access Strategy] Property is set to one of the following values: [Confluent Content-Encoded Schema Reference], [Use 'Schema Name' Property], [HWX Schema Reference Attributes], [HWX Content-Encoded Schema Reference]</strong></td></tr><tr><td id="name">Schema Name</td><td>schema-name</td><td id="default-value">${schema.name}</td><td id="allowable-values"></td><td id="description">Specifies the name of the schema to lookup in the Schema Registry property<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong><br/><br/><strong>This Property is only considered if </strong><strong>the [Schema Access Strategy] Property has a value of "Use 'Schema Name' Property".</strong></td></tr><tr><td id="name">Schema Version</td><td>schema-version</td><td></td><td id="allowable-values"></td><td id="description">Specifies the version of the schema to lookup in the Schema Registry. If not specified then the latest version of the schema will be retrieved.<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong><br/><br/><strong>This Property is only considered if </strong><strong>the [Schema Access Strategy] Property has a value of "Use 'Schema Name' Property".</strong></td></tr><tr><td id="name">Schema Branch</td><td>schema-branch</td><td></td><td id="allowable-values"></td><td id="description">Specifies the name of the branch to use when looking up the schema in the Schema Registry property. If the chosen Schema Registry does not support branching, this value will be ignored.<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong><br/><br/><strong>This Property is only considered if </strong><strong>the [Schema Access Strategy] Property has a value of "Use 'Schema Name' Property".</strong></td></tr><tr><td id="name">Schema Text</td><td>schema-text</td><td id="default-value">${avro.schema}</td><td id="allowable-values"></td><td id="description">The text of an Avro-formatted Schema<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong><br/><br/><strong>This Property is only considered if </strong><strong>the [Schema Access Strategy] Property has a value of "Use 'Schema Text' Property".</strong></td></tr><tr><td id="name"><strong>CSV Parser</strong></td><td>csv-reader-csv-parser</td><td id="default-value">Apache Commons CSV</td><td id="allowable-values"><ul><li>Apache Commons CSV <img src="../../../../../html/images/iconInfo.png" alt="The CSV parser implementation from the Apache Commons CSV library." title="The CSV parser implementation from the Apache Commons CSV library."></img></li><li>Jackson CSV <img src="../../../../../html/images/iconInfo.png" alt="The CSV parser implementation from the Jackson Dataformats library." title="The CSV parser implementation from the Jackson Dataformats library."></img></li></ul></td><td id="description">Specifies which parser to use to read CSV records. NOTE: Different parsers may support different subsets of functionality and may also exhibit different levels of performance.</td></tr><tr><td id="name">Date Format</td><td>Date Format</td><td></td><td id="allowable-values"></td><td id="description">Specifies the format to use when reading/writing Date fields. If not specified, Date fields will be assumed to be number of milliseconds since epoch (Midnight, Jan 1, 1970 GMT). If specified, the value must match the Java Simple Date Format (for example, MM/dd/yyyy for a two-digit month, followed by a two-digit day, followed by a four-digit year, all separated by '/' characters, as in 01/01/2017).</td></tr><tr><td id="name">Time Format</td><td>Time Format</td><td></td><td id="allowable-values"></td><td id="description">Specifies the format to use when reading/writing Time fields. If not specified, Time fields will be assumed to be number of milliseconds since epoch (Midnight, Jan 1, 1970 GMT). If specified, the value must match the Java Simple Date Format (for example, HH:mm:ss for a two-digit hour in 24-hour format, followed by a two-digit minute, followed by a two-digit second, all separated by ':' characters, as in 18:04:15).</td></tr><tr><td id="name">Timestamp Format</td><td>Timestamp Format</td><td></td><td id="allowable-values"></td><td id="description">Specifies the format to use when reading/writing Timestamp fields. If not specified, Timestamp fields will be assumed to be number of milliseconds since epoch (Midnight, Jan 1, 1970 GMT). If specified, the value must match the Java Simple Date Format (for example, MM/dd/yyyy HH:mm:ss for a two-digit month, followed by a two-digit day, followed by a four-digit year, all separated by '/' characters; and then followed by a two-digit hour in 24-hour format, followed by a two-digit minute, followed by a two-digit second, all separated by ':' characters, as in 01/01/2017 18:04:15).</td></tr><tr><td id="name"><strong>CSV Format</strong></td><td>CSV Format</td><td id="default-value">Custom Format</td><td id="allowable-values"><ul><li>Custom Format <img src="../../../../../html/images/iconInfo.png" alt="The format of the CSV is configured by using the properties of this Controller Service, such as Value Separator" title="The format of the CSV is configured by using the properties of this Controller Service, such as Value Separator"></img></li><li>RFC 4180 <img src="../../../../../html/images/iconInfo.png" alt="CSV data follows the RFC 4180 Specification defined at https://tools.ietf.org/html/rfc4180" title="CSV data follows the RFC 4180 Specification defined at https://tools.ietf.org/html/rfc4180"></img></li><li>Microsoft Excel <img src="../../../../../html/images/iconInfo.png" alt="CSV data follows the format used by Microsoft Excel" title="CSV data follows the format used by Microsoft Excel"></img></li><li>Tab-Delimited <img src="../../../../../html/images/iconInfo.png" alt="CSV data is Tab-Delimited instead of Comma Delimited" title="CSV data is Tab-Delimited instead of Comma Delimited"></img></li><li>MySQL Format <img src="../../../../../html/images/iconInfo.png" alt="CSV data follows the format used by MySQL" title="CSV data follows the format used by MySQL"></img></li><li>Informix Unload <img src="../../../../../html/images/iconInfo.png" alt="The format used by Informix when issuing the UNLOAD TO file_name command" title="The format used by Informix when issuing the UNLOAD TO file_name command"></img></li><li>Informix Unload Escape Disabled <img src="../../../../../html/images/iconInfo.png" alt="The format used by Informix when issuing the UNLOAD TO file_name command with escaping disabled" title="The format used by Informix when issuing the UNLOAD TO file_name command with escaping disabled"></img></li></ul></td><td id="description">Specifies which "format" the CSV data is in, or specifies if custom formatting should be used.</td></tr><tr><td id="name"><strong>Value Separator</strong></td><td>Value Separator</td><td id="default-value">,</td><td id="allowable-values"></td><td id="description">The character that is used to separate values/fields in a CSV Record. If the property has been specified via Expression Language but the expression gets evaluated to an invalid Value Separator at runtime, then it will be skipped and the default Value Separator will be used.<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong><br/><br/><strong>This Property is only considered if </strong><strong>the [CSV Format] Property has a value of "Custom Format".</strong></td></tr><tr><td id="name"><strong>Record Separator</strong></td><td>Record Separator</td><td id="default-value">\n</td><td id="allowable-values"></td><td id="description">Specifies the characters to use in order to separate CSV Records<br/><br/><strong>This Property is only considered if </strong><strong>the [CSV Format] Property has a value of "Custom Format".</strong></td></tr><tr><td id="name"><strong>Treat First Line as Header</strong></td><td>Skip Header Line</td><td id="default-value">false</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">Specifies whether or not the first line of CSV should be considered a Header or should be considered a record. If the Schema Access Strategy indicates that the columns must be defined in the header, then this property will be ignored, since the header must always be present and won't be processed as a Record. Otherwise, if 'true', then the first line of CSV data will not be processed as a record and if 'false',then the first line will be interpreted as a record.</td></tr><tr><td id="name">Ignore CSV Header Column Names</td><td>ignore-csv-header</td><td id="default-value">false</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">If the first line of a CSV is a header, and the configured schema does not match the fields named in the header line, this controls how the Reader will interpret the fields. If this property is true, then the field names mapped to each column are driven only by the configured schema and any fields not in the schema will be ignored. If this property is false, then the field names found in the CSV Header will be used as the names of the fields.</td></tr><tr><td id="name"><strong>Quote Character</strong></td><td>Quote Character</td><td id="default-value">"</td><td id="allowable-values"></td><td id="description">The character that is used to quote values so that escape characters do not have to be used. If the property has been specified via Expression Language but the expression gets evaluated to an invalid Quote Character at runtime, then it will be skipped and the default Quote Character will be used.<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong><br/><br/><strong>This Property is only considered if </strong><strong>the [CSV Format] Property has a value of "Custom Format".</strong></td></tr><tr><td id="name"><strong>Escape Character</strong></td><td>Escape Character</td><td id="default-value">\</td><td id="allowable-values"></td><td id="description">The character that is used to escape characters that would otherwise have a specific meaning to the CSV Parser. If the property has been specified via Expression Language but the expression gets evaluated to an invalid Escape Character at runtime, then it will be skipped and the default Escape Character will be used. Setting it to an empty string means no escape character should be used.<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong><br/><br/><strong>This Property is only considered if </strong><strong>the [CSV Format] Property has a value of "Custom Format".</strong></td></tr><tr><td id="name">Comment Marker</td><td>Comment Marker</td><td></td><td id="allowable-values"></td><td id="description">The character that is used to denote the start of a comment. Any line that begins with this comment will be ignored.<br/><strong>Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)</strong><br/><br/><strong>This Property is only considered if </strong><strong>the [CSV Format] Property has a value of "Custom Format".</strong></td></tr><tr><td id="name">Null String</td><td>Null String</td><td></td><td id="allowable-values"></td><td id="description">Specifies a String that, if present as a value in the CSV, should be considered a null field instead of using the literal value.<br/><br/><strong>This Property is only considered if </strong><strong>the [CSV Format] Property has a value of "Custom Format".</strong></td></tr><tr><td id="name"><strong>Trim Fields</strong></td><td>Trim Fields</td><td id="default-value">true</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">Whether or not white space should be removed from the beginning and end of fields<br/><br/><strong>This Property is only considered if </strong><strong>the [CSV Format] Property has a value of "Custom Format".</strong></td></tr><tr><td id="name"><strong>Character Set</strong></td><td>csvutils-character-set</td><td id="default-value">UTF-8</td><td id="allowable-values"></td><td id="description">The Character Encoding that is used to encode/decode the CSV file</td></tr><tr><td id="name">Allow Duplicate Header Names</td><td>csvutils-allow-duplicate-header-names</td><td id="default-value">true</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">Whether duplicate header names are allowed. Header names are case-sensitive, for example "name" and "Name" are treated as separate fields. Handling of duplicate header names is CSV Parser specific (where applicable): |
| * Apache Commons CSV - duplicate headers will result in column data "shifting" right with new fields created for "unknown_field_index_X" where "X" is the CSV column index number |
| * Jackson CSV - duplicate headers will be de-duplicated with the field value being that of the right-most duplicate CSV column<br/><br/><strong>This Property is only considered if </strong><strong>the [CSV Format] Property has a value of "Custom Format".</strong></td></tr><tr><td id="name"><strong>Trim double quote</strong></td><td>Trim double quote</td><td id="default-value">true</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">Whether or not to trim starting and ending double quotes. For example: with trim string '"test"' would be parsed to 'test', without trim would be parsed to '"test"'.If set to 'false' it means full compliance with RFC-4180. Default value is true, with trim.<br/><br/><strong>This Property is only considered if </strong><strong>the [CSV Format] Property has a value of "RFC 4180".</strong></td></tr></table><h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.<h3>System Resource Considerations:</h3>None specified.</body></html> |
