Google Git
Sign in
apache / amoro / 7f70f01458b52a73afa471e8998078c71988560c
commit7f70f01458b52a73afa471e8998078c71988560c[log] [tgz]
authorbaiyangtx <xiangnebula@163.com>Fri Aug 23 11:01:15 2024 +0800
committerGitHub <noreply@github.com>Fri Aug 23 11:01:15 2024 +0800
tree308b6d1d521cbfa346fafdc08e136292e8dbe4a0
parent8c351a06b527026c84ba5ec0b49509c634b72d89 [diff]
[AMORO-3118][Improvement]: Move iceberg codes from amoro-core to a new module amoro-iceberg (#3108)

* move mixed-format codes from core to a new module amoro-iceberg

* module refactor

* fix reviews
295 files changed
tree: 308b6d1d521cbfa346fafdc08e136292e8dbe4a0
  1. .github/
  2. .idea/
  3. amoro-ams/
  4. amoro-core/
  5. amoro-iceberg-format/
  6. amoro-mixed-format/
  7. charts/
  8. dev/
  9. docker/
  10. docs/
  11. licenses-binary/
  12. tools/
  13. .gitignore
  14. CONTRIBUTING.md
  15. DISCLAIMER
  16. LICENSE
  17. LICENSE-binary
  18. NOTICE
  19. NOTICE-binary
  20. pom.xml
  21. README.md
README.md

Apache Amoro (incubating) is a Lakehouse management system built on open data lake formats. Working with compute engines including Flink, Spark, and Trino, Amoro brings pluggable and self-managed features for Lakehouse to provide out-of-the-box data warehouse experience, and helps data platforms or products easily build infra-decoupled, stream-and-batch-fused and lake-native architecture.

Architecture

Here is the architecture diagram of Amoro:

  • AMS: Amoro Management Service provides Lakehouse management features, like self-optimizing, data expiration, etc. It also provides a unified catalog service for all compute engines, which can also be combined with existing metadata services.
  • Plugins: Amoro provides a wide selection of external plugins to meet different scenarios.
    • Optimizers: The self-optimizing execution engine plugin asynchronously performs merging, sorting, deduplication, layout optimization, and other operations on all type table format tables.
    • Terminal: SQL command-line tools, provide various implementations like local Spark and Kyuubi.
    • LogStore: Provide millisecond to second level SLAs for real-time data processing based on message queues like Kafka and Pulsar.

Supported table formats

Amoro can manage tables of different table formats, similar to how MySQL/ClickHouse can choose different storage engines. Amoro meets diverse user needs by using different table formats. Currently, Amoro supports four table formats:

  • Iceberg format: Users can directly entrust their Iceberg tables to Amoro for maintenance, so that users can not only use all the functions of Iceberg tables, but also enjoy the performance and stability improvements brought by Amoro.
  • Mixed-Iceberg format: Amoro provides a set of more optimized formats for streaming update scenarios on top of the Iceberg format. If users have high performance requirements for streaming updates or have demands for CDC incremental data reading functions, they can choose to use the Mixed-Iceberg format.
  • Mixed-Hive format: Many users do not want to affect the business originally built on Hive while using data lakes. Therefore, Amoro provides the Mixed-Hive format, which can upgrade Hive tables to Mixed-Hive format only through metadata migration, and the original Hive tables can still be used normally. This ensures business stability and benefits from the advantages of data lake computing.
  • Paimon format: Amoro supports displaying metadata information in the Paimon format, including Schema, Options, Files, Snapshots, DDLs, and Compaction information.

Supported engines

Iceberg format

Iceberg format tables use the engine integration method provided by the Iceberg community. For details, please refer to: Iceberg Docs.

Mixed format

Amoro support multiple processing engines for Mixed format as below:

Processing EngineVersionBatch ReadBatch WriteBatch OverwriteStreaming ReadStreaming WriteCreate TableAlter Table
Flink1.15.x, 1.16.x, 1.17.x✔✔✖✔✔✔✖
Spark3.1, 3.2, 3.3✔✔✔✖✖✔✔
Hive2.x, 3.x✔✖✔✖✖✖✔
Trino406✔✖✔✖✖✖✔

Features

  • Self-optimizing - Continuously optimizing tables, including compacting small files, change files, regularly delete expired files to keep high query performance and reducing storage costs.
  • Multiple Formats - Support different table formats such as Iceberg, Mixed-Iceberg and Mixed-Hive to meet different scenario requirements and provide them with unified management capabilities.
  • Catalog Service - Provide a unified catalog service for all compute engines, which can also used with existing metadata store service such as Hive Metastore and AWS Glue.
  • Rich Plugins - Provide various plugins to integrate with other systems, like continuously optimizing with Flink and data analysis with Spark and Kyuubi.
  • Management Tools - Provide a variety of management tools, including WEB UI and standard SQL command line, to help you get started faster and integrate with other systems more easily.
  • Infrastructure Independent - Can be easily deployed and used in private environments, cloud environments, hybrid cloud environments, and multi-cloud environments.

Modules

Amoro contains modules as below:

  • amoro-core contains core abstractions and common implementation for other modules
  • amoro-ams is amoro management service module
    • amoro-ams-api contains ams thrift api and common interfaces
    • amoro-ams-dashboard is the dashboard frontend for ams
    • amoro-ams-server is the backend server for ams
    • amoro-ams-optimizer provides default optimizer implementation
  • amoro-mixed-format provides Mixed format implementation
    • amoro-mixed-format-hive integrates with Apache Hive and implements Mixed Hive format
    • amoro-mixed-format-flink provides Flink connectors for Mixed format tables (use amoro-flink-runtime for a shaded version)
    • amoro-mixed-format-spark provides Spark connectors for Mixed format tables (use amoro-spark-runtime for a shaded version)
    • amoro-mixed-format-trino provides Trino connectors for Mixed format tables

Building

Amoro is built using Maven with JDK 8 and JDK 17(only for amoro-mixed-format/amoro-mixed-format-trino module).

  • Build all modules without amoro-mixed-format-trino: mvn clean package
  • Build and skip tests: mvn clean package -DskipTests
  • Build and skip dashboard: mvn clean package -Pskip-dashboard-build
  • Build and disable disk storage, RocksDB will NOT be introduced to avoid memory overflow: mvn clean package -DskipTests -Pno-extented-disk-storage
  • Build with hadoop 2.x(the default is 3.x) dependencies: mvn clean package -DskipTests -Phadoop2
  • Specify Flink version for Flink optimizer(the default is 1.20.0): mvn clean package -DskipTests -Dflink-optimizer.flink-version=1.20.0
    • If the version of Flink is below 1.15.0, you also need to add the -Pflink-optimizer-pre-1.15 parameter: mvn clean package -DskipTests -Pflink-optimizer-pre-1.15 -Dflink-optimizer.flink-version=1.14.6
  • Specify Spark version for Spark optimizer(the default is 3.3.3): mvn clean package -DskipTests -Dspark-optimizer.spark-version=3.3.3
  • Build amoro-mixed-format-trino module under JDK 17: mvn clean package -DskipTests -Pformat-mixed-format-trino,build-mixed-format-trino -pl 'amoro-mixed-format/amoro-mixed-format-trino' -am.
  • Build all modules: mvn clean package -DskipTests -Ptoolchain,build-mixed-format-trino, besides you need config toolchains.xml in ${user.home}/.m2/ dir with content below.
<?xml version="1.0" encoding="UTF-8"?>
<toolchains>
    <toolchain>
        <type>jdk</type>
        <provides>
            <version>17</version>
            <vendor>sun</vendor>
        </provides>
        <configuration>
            <jdkHome>${YourJDK17Home}</jdkHome>
        </configuration>
    </toolchain>
</toolchains>

Quickstart

Visit https://amoro.apache.org/quick-demo/ to quickly explore what amoro can do.

Join Community

If you are interested in Lakehouse, Data Lake Format, welcome to join our community, we welcome any organizations, teams and individuals to grow together, and sincerely hope to help users better use Data Lake Format through open source.

Join the Amoro WeChat Group: Add " kllnn999 " as a friend on WeChat and specify “Amoro lover”.

Contributors

This project exists thanks to all the people who contribute.

Made with contrib.rocks.

Star History

Star History Chart