| commit | f8b7e16ec0c2072846ab41c098cf17e3dbbb38b2 | [log] [tgz] |
|---|---|---|
| author | chendapao <feloxx@163.com> | Mon Nov 27 14:46:00 2023 +0800 |
| committer | GitHub <noreply@github.com> | Mon Nov 27 14:46:00 2023 +0800 |
| tree | 1a90cdd936dd73c5ccb9207a1b3b892b5d088c43 | |
| parent | efdcb5690efd8f705ab3f2603b8e85d67a6ed0c7 [diff] |
Add paimon.query-pushdown-enabled to readme (#25)
This repository is Presto Connector for the Apache Paimon project.
Apache Paimon is an open source project of The Apache Software Foundation (ASF).
mvn clean install -DskipTests
During the packaging process, you may encounter the following errors:
[ERROR] Failed to execute goal on project paimon-presto: Could not resolve dependencies for project org.apache.paimon:paimon-presto:pom:0.6-SNAPSHOT: The following artifacts could not be resolved: org.apache.paimon:paimon-bundle:jar:0.6-SNAPSHOT (absent): Could not find artifact org.apache.paimon:paimon-bundle:jar:0.6-SNAPSHOT in xxx
You can resolve the packaging issue by adding the following Maven repository addresses to your settings.xml or to the pom.xml of the current project:
<repositories> <repository> <id>apache-releases</id> <name>apache releases</name> <url>https://repository.apache.org/content/repositories/releases/</url> </repository> <repository> <id>apache-snapshots</id> <name>apache snapshots</name> <url>https://repository.apache.org/content/repositories/snapshots/</url> </repository> </repositories>
After the packaging is complete, you can choose the corresponding connector based on your own Presto version:
| Version | Package |
|---|---|
| [0.236, 0.268) | ./paimon-presto-0.236/target/paimon-presto-0.236-0.6-SNAPSHOT-plugin.tar.gz |
| [0.268, 0.273) | ./paimon-presto-0.268/target/paimon-presto-0.268-0.6-SNAPSHOT-plugin.tar.gz |
| [0.273, latest] | ./paimon-presto-0.273/target/paimon-presto-0.273-0.6-SNAPSHOT-plugin.tar.gz |
Of course, we also support different versions of Hive and Hadoop. But note that we utilize Presto-shaded versions of Hive and Hadoop packages to address dependency conflicts. You can check the following two links to select the appropriate versions of Hive and Hadoop:
Both Hive 2 and 3, as well as Hadoop 2 and 3, are supported.
For example, if your presto version is 0.274, hive and hadoop version is 2.x, you could run:
mvn clean install -DskipTests -am -pl paimon-presto-0.273 -Dpresto.version=0.274 -Dhadoop.apache2.version=2.7.4-9 -Dhive.apache.version=1.2.2-2
tar -zxf paimon-presto-${PRESTO_VERSION}/target/paimon-presto-${PRESTO_VERSION}-${PAIMON_VERSION}-plugin.tar.gz -C ${PRESTO_HOME}/plugin
Note that, the variable PRESTO_VERSION is module name, must be one of 0.236, 0.268, 0.273.
cd ${PRESTO_HOME} mkdir -p etc/catalog
Query FileSystem table:
vim etc/catalog/paimon.properties
and set the following config:
connector.name=paimon # set your filesystem path, such as hdfs://namenode01:8020/path and s3://${YOUR_S3_BUCKET}/path warehouse=${YOUR_FS_PATH} # Enable paimon query pushdown. paimon.query-pushdown-enabled=true
If you are using HDFS FileSystem, you will also need to do one more thing: choose one of the following ways to configure your HDFS:
hadoop-conf-dir in the properties.If you are using S3 FileSystem, you need to add paimon-s3-${PAIMON_VERSION}.jar in ${PRESTO_HOME}/plugin/paimon and additionally configure the following properties in paimon.properties:
s3.endpoint=${YOUR_ENDPOINTS} s3.access-key=${YOUR_AK} s3.secret-key=${YOUR_SK}
Query HiveCatalog table:
vim etc/catalog/paimon.properties
and set the following config:
connector.name=paimon # set your filesystem path, such as hdfs://namenode01:8020/path and s3://${YOUR_S3_BUCKET}/path warehouse=${YOUR_FS_PATH} metastore=hive uri=thrift://${YOUR_HIVE_METASTORE}:9083