components/camel-csv/src/main/docs/csv-dataformat.adoc

[[csv-dataformat]]
= CSV DataFormat

*Since Camel 1.3*

The CSV Data Format uses
http://commons.apache.org/proper/commons-csv/[Apache Commons CSV] to
handle CSV payloads (Comma Separated Values) such as those
exported/imported by Excel.


== Options

// dataformat options: START
The CSV dataformat supports 29 options, which are listed below.



[width="100%",cols="2s,1m,1m,6",options="header"]
|===
| Name | Default | Java Type | Description
| formatRef |  | String | The reference format to use, it will be updated with the other format options, the default value is CSVFormat.DEFAULT
| formatName |  | String | The name of the format to use, the default value is CSVFormat.DEFAULT
| commentMarkerDisabled | false | Boolean | Disables the comment marker of the reference format.
| commentMarker |  | String | Sets the comment marker of the reference format.
| delimiter |  | String | Sets the delimiter to use. The default value is , (comma)
| escapeDisabled | false | Boolean | Use for disabling using escape character
| escape |  | String | Sets the escape character to use
| headerDisabled | false | Boolean | Use for disabling headers
| header |  | List | To configure the CSV headers
| allowMissingColumnNames | false | Boolean | Whether to allow missing column names.
| ignoreEmptyLines | false | Boolean | Whether to ignore empty lines.
| ignoreSurroundingSpaces | false | Boolean | Whether to ignore surrounding spaces
| nullStringDisabled | false | Boolean | Used to disable null strings
| nullString |  | String | Sets the null string
| quoteDisabled | false | Boolean | Used to disable quotes
| quote |  | String | Sets the quote which by default is
| recordSeparatorDisabled |  | String | Used for disabling record separator
| recordSeparator |  | String | Sets the record separator (aka new line) which by default is new line characters (CRLF)
| skipHeaderRecord | false | Boolean | Whether to skip the header record in the output
| quoteMode |  | String | Sets the quote mode
| ignoreHeaderCase | false | Boolean | Sets whether or not to ignore case when accessing header names.
| trim | false | Boolean | Sets whether or not to trim leading and trailing blanks.
| trailingDelimiter | false | Boolean | Sets whether or not to add a trailing delimiter.
| marshallerFactoryRef |  | String | Sets the implementation of the CsvMarshallerFactory interface which is able to customize marshalling/unmarshalling behavior by extending CsvMarshaller or creating it from scratch.
| lazyLoad | false | Boolean | Whether the unmarshalling should produce an iterator that reads the lines on the fly or if all the lines must be read at one.
| useMaps | false | Boolean | Whether the unmarshalling should produce maps (HashMap)for the lines values instead of lists. It requires to have header (either defined or collected).
| useOrderedMaps | false | Boolean | Whether the unmarshalling should produce ordered maps (LinkedHashMap) for the lines values instead of lists. It requires to have header (either defined or collected).
| recordConverterRef |  | String | Refers to a custom CsvRecordConverter to lookup from the registry to use.
| contentTypeHeader | false | Boolean | Whether the data format should set the Content-Type header with the type from the data format if the data format is capable of doing so. For example application/xml for data formats marshalling to XML, or application/json for data formats marshalling to JSon etc.
|===
// dataformat options: END
// spring-boot-auto-configure options: START
== Spring Boot Auto-Configuration

When using Spring Boot make sure to use the following Maven dependency to have support for auto configuration:

[source,xml]
----
<dependency>
  <groupId>org.apache.camel.springboot</groupId>
  <artifactId>camel-csv-starter</artifactId>
  <version>x.x.x</version>
  <!-- use the same version as your Camel core version -->
</dependency>
----


The component supports 30 options, which are listed below.



[width="100%",cols="2,5,^1,2",options="header"]
|===
| Name | Description | Default | Type
| *camel.dataformat.csv.allow-missing-column-names* | Whether to allow missing column names. | false | Boolean
| *camel.dataformat.csv.comment-marker* | Sets the comment marker of the reference format. |  | String
| *camel.dataformat.csv.comment-marker-disabled* | Disables the comment marker of the reference format. | false | Boolean
| *camel.dataformat.csv.content-type-header* | Whether the data format should set the Content-Type header with the type from the data format if the data format is capable of doing so. For example application/xml for data formats marshalling to XML, or application/json for data formats marshalling to JSon etc. | false | Boolean
| *camel.dataformat.csv.delimiter* | Sets the delimiter to use. The default value is , (comma) |  | String
| *camel.dataformat.csv.enabled* | Whether to enable auto configuration of the csv data format. This is enabled by default. |  | Boolean
| *camel.dataformat.csv.escape* | Sets the escape character to use |  | String
| *camel.dataformat.csv.escape-disabled* | Use for disabling using escape character | false | Boolean
| *camel.dataformat.csv.format-name* | The name of the format to use, the default value is CSVFormat.DEFAULT |  | String
| *camel.dataformat.csv.format-ref* | The reference format to use, it will be updated with the other format options, the default value is CSVFormat.DEFAULT |  | String
| *camel.dataformat.csv.header* | To configure the CSV headers |  | List
| *camel.dataformat.csv.header-disabled* | Use for disabling headers | false | Boolean
| *camel.dataformat.csv.ignore-empty-lines* | Whether to ignore empty lines. | false | Boolean
| *camel.dataformat.csv.ignore-header-case* | Sets whether or not to ignore case when accessing header names. | false | Boolean
| *camel.dataformat.csv.ignore-surrounding-spaces* | Whether to ignore surrounding spaces | false | Boolean
| *camel.dataformat.csv.lazy-load* | Whether the unmarshalling should produce an iterator that reads the lines on the fly or if all the lines must be read at one. | false | Boolean
| *camel.dataformat.csv.marshaller-factory-ref* | Sets the implementation of the CsvMarshallerFactory interface which is able to customize marshalling/unmarshalling behavior by extending CsvMarshaller or creating it from scratch. |  | String
| *camel.dataformat.csv.null-string* | Sets the null string |  | String
| *camel.dataformat.csv.null-string-disabled* | Used to disable null strings | false | Boolean
| *camel.dataformat.csv.quote* | Sets the quote which by default is |  | String
| *camel.dataformat.csv.quote-disabled* | Used to disable quotes | false | Boolean
| *camel.dataformat.csv.quote-mode* | Sets the quote mode |  | String
| *camel.dataformat.csv.record-converter-ref* | Refers to a custom CsvRecordConverter to lookup from the registry to use. |  | String
| *camel.dataformat.csv.record-separator* | Sets the record separator (aka new line) which by default is new line characters (CRLF) |  | String
| *camel.dataformat.csv.record-separator-disabled* | Used for disabling record separator |  | String
| *camel.dataformat.csv.skip-header-record* | Whether to skip the header record in the output | false | Boolean
| *camel.dataformat.csv.trailing-delimiter* | Sets whether or not to add a trailing delimiter. | false | Boolean
| *camel.dataformat.csv.trim* | Sets whether or not to trim leading and trailing blanks. | false | Boolean
| *camel.dataformat.csv.use-maps* | Whether the unmarshalling should produce maps (HashMap)for the lines values instead of lists. It requires to have header (either defined or collected). | false | Boolean
| *camel.dataformat.csv.use-ordered-maps* | Whether the unmarshalling should produce ordered maps (LinkedHashMap) for the lines values instead of lists. It requires to have header (either defined or collected). | false | Boolean
|===
// spring-boot-auto-configure options: END
ND



== Marshalling a Map to CSV

The component allows you to marshal a Java Map (or any other message
type that can be converted in a Map) into a
CSV payload.

Considering the following body 

[source,java]
-------------------------------------------------------
Map<String, Object> body = new LinkedHashMap<>();
body.put("foo", "abc");
body.put("bar", 123);
-------------------------------------------------------

and this Java route definition 

[source,java]
-------------------------------------------------------
from("direct:start")
    .marshal().csv()
    .to("mock:result");
-------------------------------------------------------

or this XML route definition 

[source,xml]
-------------------------------------------------------
<route>
    <from uri="direct:start" />
    <marshal>
        <csv />
    </marshal>
    <to uri="mock:result" />
</route>
-------------------------------------------------------


then it will produce 

----
abc,123
----

== Unmarshalling a CSV message into a Java List

Unmarshalling will transform a CSV messsage into a Java List with CSV
file lines (containing another List with all the field values).

An example: we have a CSV file with names of persons, their IQ and their
current activity.

[source,text]
-----------------------------------------------------
Jack Dalton, 115, mad at Averell
Joe Dalton, 105, calming Joe
William Dalton, 105, keeping Joe from killing Averell
Averell Dalton, 80, playing with Rantanplan
Lucky Luke, 120, capturing the Daltons
-----------------------------------------------------

We can now use the CSV component to unmarshal this file:

[source,java]
---------------------------------------------------------------
from("file:src/test/resources/?fileName=daltons.csv&noop=true")
    .unmarshal().csv()
    .to("mock:daltons");
---------------------------------------------------------------

The resulting message will contain a `List<List<String>>` like...

[source,java]
--------------------------------------------------------------------------------------------------------------
List<List<String>> data = (List<List<String>>) exchange.getIn().getBody();
for (List<String> line : data) {
    LOG.debug(String.format("%s has an IQ of %s and is currently %s", line.get(0), line.get(1), line.get(2)));
}
--------------------------------------------------------------------------------------------------------------

== Marshalling a List<Map> to CSV

*Since Camel 2.1*

If you have multiple rows of data you want to be marshalled into CSV
format you can now store the message payload as a
`List<Map<String, Object>>` object where the list contains a Map for
each row.

== File Poller of CSV, then unmarshaling

Given a bean which can handle the incoming data...

*MyCsvHandler.java*

[source,java]
-------------------------------------------------------
// Some comments here
public void doHandleCsvData(List<List<String>> csvData)
{
    // do magic here
}
-------------------------------------------------------

... your route then looks as follows

[source,xml]
------------------------------------------------------------------------------------------------
<route>
        <!-- poll every 10 seconds -->
        <from uri="file:///some/path/to/pickup/csvfiles?delete=true&amp;delay=10000" />
        <unmarshal><csv /></unmarshal>
        <to uri="bean:myCsvHandler?method=doHandleCsvData" />
</route>
------------------------------------------------------------------------------------------------

== Marshaling with a pipe as delimiter
Considering the following body

[source,java]
-------------------------------------------------------
Map<String, Object> body = new LinkedHashMap<>();
body.put("foo", "abc");
body.put("bar", 123);
------------------------------------------------------- 


and this Java route definition 

[source,java]
-------------------------------------------------------
from("direct:start")
    .marshal(new CsvDataFormat().setDelimiter(&#39;|&#39;))
    .to("mock:result")
------------------------------------------------------- 

or this XML route definition 

[source,xml]
-------------------------------------------------------
<route>
  <from uri="direct:start" />
  <marshal>
    <csv delimiter="|" />
  </marshal>
  <to uri="mock:result" />
</route>
------------------------------------------------------- 

then it will produce 

-------------------------------------------------------
abc|123
------------------------------------------------------- 

[[CSV-UsingautogenColumns,configRefandstrategyRefattributesinsideXMLDSL]]
Using autogenColumns, configRef and strategyRef attributes inside XML
== DSL

*Since Camel 2.9.2 / 2.10 and deleted for Camel 2.15*

You can customize the CSV Data Format to make use
of your own `CSVConfig` and/or `CSVStrategy`. Also note that the default
value of the `autogenColumns` option is true. The following example
should illustrate this customization.

[source,xml]
-----------------------------------------------------------------------------------------------------------------------------
<route>
  <from uri="direct:start" />
  <marshal>
    <!-- make use of a strategy other than the default one which is 'org.apache.commons.csv.CSVStrategy.DEFAULT_STRATEGY' -->
    <csv autogenColumns="false" delimiter="|" configRef="csvConfig" strategyRef="excelStrategy" />
  </marshal>
  <convertBodyTo type="java.lang.String" />
  <to uri="mock:result" />
</route>

<bean id="csvConfig" class="org.apache.commons.csv.writer.CSVConfig">
  <property name="fields">
    <list>
      <bean class="org.apache.commons.csv.writer.CSVField">
        <property name="name" value="orderId" />
      </bean>
      <bean class="org.apache.commons.csv.writer.CSVField">
        <property name="name" value="amount" />
      </bean>
    </list>
  </property>
</bean>

<bean id="excelStrategy" class="org.springframework.beans.factory.config.FieldRetrievingFactoryBean">
  <property name="staticField" value="org.apache.commons.csv.CSVStrategy.EXCEL_STRATEGY" />
</bean>
-----------------------------------------------------------------------------------------------------------------------------

== Using skipFirstLine or skipHeaderRecord option while unmarshaling

*For Camel >= 2.16.5 
The instruction for CSV Data format to skip headers or first line is the following. 
Usign the Spring/XML DSL:

[source,xml]
---------------------------------------------------
<route>
  <from uri="direct:start" />
  <unmarshal>
    <csv skipHeaderRecord="true" />
  </unmarshal>
  <to uri="bean:myCsvHandler?method=doHandleCsv" />
</route>
---------------------------------------------------

*Since Camel 2.10 and deleted for Camel 2.15*

You can instruct the CSV Data Format to skip the
first line which contains the CSV headers. Using the Spring/XML DSL:

[source,xml]
---------------------------------------------------
<route>
  <from uri="direct:start" />
  <unmarshal>
    <csv skipFirstLine="true" />
  </unmarshal>
  <to uri="bean:myCsvHandler?method=doHandleCsv" />
</route>
---------------------------------------------------

Or the Java DSL:

[source,java]
--------------------------------------------
CsvDataFormat csv = new CsvDataFormat();
csv.setSkipFirstLine(true);

from("direct:start")
  .unmarshal(csv)
.to("bean:myCsvHandler?method=doHandleCsv");
--------------------------------------------

== Unmarshaling with a pipe as delimiter

Using the Spring/XML DSL:

[source,xml]
---------------------------------------------------
<route>
  <from uri="direct:start" />
  <unmarshal>
    <csv delimiter="|" />
  </unmarshal>
  <to uri="bean:myCsvHandler?method=doHandleCsv" />
</route>
---------------------------------------------------

Or the Java DSL:

[source,java]
----------------------------------------------------
CsvDataFormat csv = new CsvDataFormat();
CSVStrategy strategy = CSVStrategy.DEFAULT_STRATEGY;
strategy.setDelimiter('|');
csv.setStrategy(strategy);

from("direct:start")
  .unmarshal(csv)
  .to("bean:myCsvHandler?method=doHandleCsv");
----------------------------------------------------

[source,java]
----------------------------------------------
CsvDataFormat csv = new CsvDataFormat();
csv.setDelimiter("|");

from("direct:start")
  .unmarshal(csv)
  .to("bean:myCsvHandler?method=doHandleCsv");
----------------------------------------------

[source,java]
----------------------------------------------
CsvDataFormat csv = new CsvDataFormat();
CSVConfig csvConfig = new CSVConfig();
csvConfig.setDelimiter(";");
csv.setConfig(csvConfig);

from("direct:start")
  .unmarshal(csv)
  .to("bean:myCsvHandler?method=doHandleCsv");
----------------------------------------------

*Issue in CSVConfig*

It looks like that

[source,java]
--------------------------------------
CSVConfig csvConfig = new CSVConfig();
csvConfig.setDelimiter(';');
--------------------------------------

doesn't work. You have to set the delimiter as a String!

== Dependencies

To use CSV in your Camel routes you need to add a dependency on
*camel-csv*, which implements this data format.

If you use Maven you can just add the following to your pom.xml,
substituting the version number for the latest and greatest release (see
the download page for the latest versions).

[source,xml]
-------------------------------------
<dependency>
  <groupId>org.apache.camel</groupId>
  <artifactId>camel-csv</artifactId>
  <version>x.x.x</version>
</dependency>
-------------------------------------