blob: 8b90139d1e52a9b6e5379a0ae8cc7c62d4755369 [file] [log] [blame]
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<chapter id="Working with Ofbiz Data files" version="5.0" xsi:schemaLocation="http://www.docbook.org/xml/5.0b3/xsd/docbook.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xl="http://www.w3.org/1999/xlink" xmlns:xi="http://www.w3.org/2001/XInclude" xmlns="http://docbook.org/ns/docbook">
<title>The OFBiz Datafiles</title>
<section>
<section id="Intoduction">
<title>Intoduction</title>
<para>There is a data import tool in OFBiz called the DataFile tool. It uses XML files that describe flat file formats (including character delimited, fixed width, etc) and parses the flat files based on those definitions. So, by design it is somewhat like the Entity Engine. It uses a generic object to represent a row in the flat file. It includes features like a line type code for each line and can support hierarchical flat files (ie where parent/child relationships are implied by sub-records).
</para>
<para>The code is in the ofbiz/framework/datafile directory, and there is an XSD there to describe how the data file definition XML file should look.
</para>
<para>The DataFile has a web page in WebTools to parse data files and generate entity xml files (that can be imported in OFBiz) from them, and to do in/outtesting.
</para>
<para>both the import of fixed width flat files and character delimited files are implemented.
</para>
</section>
<section id="Definitions">
<title>Definitions</title>
<para>Definition file*: this is an xml file that contains one or more "Data File Definition"s; the grammar of the file is defined in framework/datafile/dtd/datafile.xsd
</para>
<para>Data File Definition*: the definition of a data file structure (e.g. names/types of the fields/columns in a "Data File"; the same "Data File" could have more than one "Data File Definition"s (e.g. the definitions could consider different subsets of all the fields available in the data file).
</para>
<para>Data file*: the text file that contains all the data (a fixed width or a character delimited text file)
</para>
</section>
<section>
<title>How to use the "Work With Data Files" screen in the Webtools application</title>
<procedure>
<title>How to use the "Work With Data Files" screen in the Webtools application</title>_Prerequisites_: a definition file (containing the fields' definition of the data file) and a data file (containing the data you want to parse/import) should be available in the OFBiz server.Steps:
<step performance="required">
<para>
connect to the Webtools application
</para>
</step>
<step performance="required">
<para>go to the "Work With Data Files" screen</para>
</step>
<step performance="required">
<para>
enter the path to the your definition file in the "Definition Filename or URL" input field
</para>
</step>
<step performance="required">
<para>
click on the submit button>
</para>
</step>
<step performance="required">
<para>
the "Data File Definition Name" input field will be changed into a drop down box containing all the definitions available in the definition file
</para>
</step>
<step performance="required">
<para>
select the definition you want to use from the drop down box>
</para>
</step>
<step performance="required">
<para>
enter the path to the your data file in the "Data Filename or URL" input field
</para>
</step>
<step performance="required">
<para>
if you want to generate an entity xml file that can be later imported in OFBiz,
enter a path for it in the "Save to entity xml file:
</para>
</step>
<step performance="required">
<para>
input field and click on the submit button; the file will be created
(see the paragraph "Tips to create Entity Xml Files" for more details)</para>
</step>
</procedure>
</section>
<section>The field.name attribute must contain the name of the entity field in which the records will be imported
(for example<programlisting><![CDATA[:<field name="productId" type="String">
</field>
]]></programlisting>
</section>
<section>
<title>Examples</title>
<section>
<example>
<title>An example of fixed width flat file import.</title>Sample fixed width CSV file posreport.csv to be imported:
<programlisting><![CDATA[
021196033702 ,5031BB GLITTER GLUE PENS BRIGH ,1 ,5031BB , 1, 299,
021196043121 ,BB4312 WONDERFOAM ASSORTED ,1 ,BB4312 , 1, 280,
021196055025 ,9905BB PLUMAGE MULTICOLOURED ,1 ,9905BB , 4, 396,
]]></programlisting>
</example>
</section>
<section>
<title>Sample xml definition file for importing select columns</title>
<example>
<title> posschema.xml:</title>
<programlisting><![CDATA[
<data-files xsi:noNamespaceSchemaLocation="http://ofbiz.apache.org/dtds/datafiles.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<data-file name="posreport" separator-style="fixed-length" type-code="text">
<record name="tillentry" limit="many">
<field name="tillCode" type="String" length="16" position="0">
</field>
<field name="name" type="String" length="32" position="17">
</field>
<field name="prodCode" type="String" length="12" position="63">
</field>
<field name="quantity" type="String" length="8" position="76">
</field>
<field name="totalPrice" type="String" length="8" position="85">
</field>
</record>
</data-file>
</data-files>
]]></programlisting>
</example>
The types listed in this sample are simple String's but the usual types are available such as Date, Long etc
</section>
<section>
<example>
<title>Another example reading fixed record little endian binary files</title>
<programlisting><![CDATA[
<data-files xsi:noNamespaceSchemaLocation="http://ofbiz.apache.org/dtds/datafiles.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<data-file name="stockdata" separator-style="fixed-record" type-code="text" record-length="768">
<record name="stockdataitem" limit="many">
<field name="barcode" type="NullTerminatedString" length="12" position="0">
</field>
<field name="prodCode" type="NullTerminatedString" length="12" position="68">
</field>
<field name="price" type="LEInteger" length="4" position="80">
</field>
<field name="name" type="NullTerminatedString" length="30" position="16">
</field>
</record>
</data-file>
</data-files>
]]></programlisting>
</example>
<procedure>
<title>In the interface enter something like:</title>
<step performance="required">
<para>
Definition Filename or URL: posschema.xml
</para>
</step>
<step performance="required">
<para>
Data File Definition Name: posreport
</para>
</step>
<step performance="required">
<para>
Data Filename or URL: posreport.csv
</para>
</step>
</procedure>
</section>
</section>
<section>
<example>
<title>Sample xml definition file for importing select columns posschema.xml:</title>
<programlisting><![CDATA[
<data-files xsi:noNamespaceSchemaLocation="http://ofbiz.apache.org/dtds/datafiles.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<data-file name="posreport" separator-style="fixed-length" type-code="text">
<record name="tillentry" limit="many">
<field name="tillCode" type="String" length="16" position="0">
</field>
<field name="name" type="String" length="32" position="17">
</field>
<field name="prodCode" type="String" length="12" position="63">
</field>
<field name="quantity" type="String" length="8" position="76">
</field>
<field name="totalPrice" type="String" length="8" position="85">
</field>
</record>
</data-file>
</data-files>
]]></programlisting>
</example>
In the interface enter something like:* Definition Filename or URL: posschema.xml* Data File Definition Name: posreport* Data Filename or URL: posreport.csvThe types listed in this sample are simple String's but the usual types are available such as Date, Long etc.
</section>
<section>
<example>
<title> Another example reading fixed record little endian binary files</title>
<programlisting><![CDATA[
<data-files xsi:noNamespaceSchemaLocation="http://ofbiz.apache.org/dtds/datafiles.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<data-file name="stockdata" separator-style="fixed-record" type-code="text" record-length="768">
<record name="stockdataitem" limit="many">
<field name="barcode" type="NullTerminatedString" length="12" position="0">
</field>
<field name="prodCode" type="NullTerminatedString" length="12" position="68">
</field>
<field name="price" type="LEInteger" length="4" position="80">
</field>
<field name="name" type="NullTerminatedString" length="30" position="16">
</field>
</record>
</data-file>
</data-files>
]]></programlisting>
</example>
</section>
</section>
</chapter>