| [[tokenize-language]] |
| = Tokenize Language |
| :page-source: core/camel-base/src/main/docs/tokenize-language.adoc |
| |
| *Available as of Camel version 2.0* |
| |
| The tokenizer language is a built-in language in camel-core, which is |
| most often used only with the Splitter EIP to split |
| a message using a token-based strategy. + |
| The tokenizer language is intended to tokenize text documents using a |
| specified delimiter pattern. It can also be used to tokenize XML |
| documents with some limited capability. For a truly XML-aware |
| tokenization, the use of the XMLTokenizer |
| language is recommended as it offers a faster, more efficient |
| tokenization specifically for XML documents. For more details |
| see Splitter. |
| |
| == Tokenize Options |
| |
| // language options: START |
| The Tokenize language supports 11 options, which are listed below. |
| |
| |
| |
| [width="100%",cols="2,1m,1m,6",options="header"] |
| |=== |
| | Name | Default | Java Type | Description |
| | token | | String | The (start) token to use as tokenizer, for example you can use the new line token. You can use simple language as the token to support dynamic tokens. |
| | endToken | | String | The end token to use as tokenizer if using start/end token pairs. You can use simple language as the token to support dynamic tokens. |
| | inheritNamespaceTagName | | String | To inherit namespaces from a root/parent tag name when using XML You can use simple language as the tag name to support dynamic names. |
| | headerName | | String | Name of header to tokenize instead of using the message body. |
| | regex | false | Boolean | If the token is a regular expression pattern. The default value is false |
| | xml | false | Boolean | Whether the input is XML messages. This option must be set to true if working with XML payloads. |
| | includeTokens | false | Boolean | Whether to include the tokens in the parts when using pairs The default value is false |
| | group | | String | To group N parts together, for example to split big files into chunks of 1000 lines. You can use simple language as the group to support dynamic group sizes. |
| | groupDelimiter | | String | Sets the delimiter to use when grouping. If this has not been set then token will be used as the delimiter. |
| | skipFirst | false | Boolean | To skip the very first element |
| | trim | true | Boolean | Whether to trim the value to remove leading and trailing whitespaces and line breaks |
| |=== |
| // language options: END |
