tree: 5f8721afb922f5276a1b00167108414a8c755765 [path history] [tgz]
  1. src/
  2. README.md
  3. REVIEWERS.md
  4. pom.xml
parquet-avro/README.md

Apache Avro integration

TODO: Add description and examples how to use parquet-avro

Available options via Hadoop Configuration

Configuration for reading

NameTypeDescription
parquet.avro.data.supplierClassThe implementation of the interface org.apache.parquet.avro.AvroDataSupplier. Available implementations in the library: GenericDataSupplier, ReflectDataSupplier, SpecificDataSupplier.
The default value is org.apache.parquet.avro.SpecificDataSupplier
parquet.avro.read.schemaStringThe Avro schema to be used for reading. It shall be compatible with the file schema. The file schema will be used directly if not set.
parquet.avro.projectionStringThe Avro schema to be used for projection.
parquet.avro.compatiblebooleanFlag for compatibility mode. true for materializing Avro IndexedRecord objects, false for materializing the related objects for either generic, specific, or reflect records.
The default value is true.

Configuration for writing

NameTypeDescription
parquet.avro.write.data.supplierClassThe implementation of the interface org.apache.parquet.avro.AvroDataSupplier. Available implementations in the library: GenericDataSupplier, ReflectDataSupplier, SpecificDataSupplier.
The default value is org.apache.parquet.avro.SpecificDataSupplier
parquet.avro.schemaStringThe Avro schema to be used for generating the Parquet schema of the file.
parquet.avro.write-old-list-structurebooleanFlag whether to write list structures in the old way (2 levels) or the new one (3 levels). When writing at 2 levels no null values are available at the element level.
The default value is true
parquet.avro.add-list-element-recordsbooleanFlag whether to assume that any repeated element in the schema is a list element.
The default value is true.
parquet.avro.write-parquet-uuidbooleanFlag whether to write the Parquet UUID logical type in case of an Avro UUID type is present.
The default value is false.