PXF is installed with connectors to Azure Blob Storage, Azure Data Lake Storage Gen2, Google Cloud Storage, AWS, MinIO, and Dell ECS S3-compatible object stores.
Before working with object store data using PXF, ensure that:
The PXF object store connectors provide built-in profiles to support the following data formats:
The PXF connectors to Azure expose the following profiles to read, and in many cases write, these supported data formats.
Note: ADL support has been deprecated. Use the ABFSS profile instead.
| Data Format | Azure Blob Storage | Azure Data Lake Storage Gen2 | Supported Operations |
|---|---|---|---|
| delimited single line plain text | wasbs:text | abfss:text | Read, Write |
| delimited single line comma-separated values of plain text | wasbs:csv | abfss:csv | Read, Write |
| multi-byte or multi-character delimited single line csv | wasbs:csv | abfss:csv | Read |
| delimited text with quoted linefeeds | wasbs:text:multi | abfss:text:multi | Read |
| fixed width single line text | wasbs:fixedwidth | abfss:fixedwidth | Read, Write |
| Avro | wasbs:avro | abfss:avro | Read, Write |
| JSON | wasbs:json | abfss:json | Read, Write |
| ORC | wasbs:orc | abfss:orc | Read, Write |
| Parquet | wasbs:parquet | abfss:parquet | Read, Write |
| AvroSequenceFile | wasbs:AvroSequenceFile | abfss:AvroSequenceFile | Read, Write |
| SequenceFile | wasbs:SequenceFile | abfss:SequenceFile | Read, Write |
Similarly, the PXF connectors to Google Cloud Storage, and S3-compatible object stores expose these profiles:
| Data Format | Google Cloud Storage | AWS S3, MinIO, or Dell ECS | Supported Operations |
|---|---|---|---|
| delimited single line plain text | gs:text | s3:text | Read, Write |
| delimited single line comma-separated values of plain text | gs:csv | s3:csv | Read, Write |
| multi-byte or multi-character delimited single line comma-separated values csv | gs:csv | s3:csv | Read |
| delimited text with quoted linefeeds | gs:text:multi | s3:text:multi | Read |
| fixed width single line text | gs:fixedwidth | s3:fixedwidth | Read, Write |
| Avro | gs:avro | s3:avro | Read, Write |
| JSON | gs:json | s3:json | Read |
| ORC | gs:orc | s3:orc | Read, Write |
| Parquet | gs:parquet | s3:parquet | Read, Write |
| AvroSequenceFile | gs:AvroSequenceFile | s3:AvroSequenceFile | Read, Write |
| SequenceFile | gs:SequenceFile | s3:SequenceFile | Read, Write |
You provide the profile name when you specify the pxf protocol on a CREATE EXTERNAL TABLE command to create an Apache Cloudberry external table that references a file or directory in the specific object store.
Note: When you create an external table that references a file or directory in an object store, you must specify a
SERVERin theLOCATIONURI.
The following command creates an external table that references a text file on S3. It specifies the profile named s3:text and the server configuration named s3srvcfg:
The following command creates an external table that references a text file on Azure Blob Storage. It specifies the profile named wasbs:text and the server configuration named wasbssrvcfg. You would provide the Azure Blob Storage container identifier and your Azure Blob Storage account name.
The following command creates an external table that references a text file on Azure Data Lake Storage Gen2. It specifies the profile named abfss:text and the server configuration named abfsssrvcfg. You would provide your Azure Data Lake Storage Gen2 account name.
The following command creates an external table that references a JSON file on Google Cloud Storage. It specifies the profile named gs:json and the server configuration named gcssrvcfg: