Google Git
Sign in
apache / kylin / refs/heads/kylin4_on_cloud
commite630a46dfbbe2170774a87e811c5840bfc10f083[log] [tgz]
authorTengting Xu <34978943+Mukvin@users.noreply.github.com>Tue Apr 12 16:48:20 2022 +0800
committerGitHub <noreply@github.com>Tue Apr 12 16:48:20 2022 +0800
tree1e31f6a589b86d2ad5b36877acc0f1160bda83a9
parentd94583fe7bc5763db028c61453efb863e9b659f8 [diff]
# minor fix log (#1856)

* # minor fix log

* # minor fix, enhance for checking stack status

* # minor fix log

* # minor fix, enhance for checking stack status

* # minor fix file path check

* # minor fix, correct spark package for kylin
5 files changed
tree: 1e31f6a589b86d2ad5b36877acc0f1160bda83a9
  1. backup/
  2. bin/
  3. client/
  4. cloudformation_templates/
  5. clouds/
  6. constant/
  7. images/
  8. instances/
  9. logs/
  10. readme/
  11. utils/
  12. .gitignore
  13. __init__.py
  14. deploy.py
  15. engine.py
  16. kylin_configs.yaml
  17. logging.ini
  18. README.md
  19. requirements.txt
README.md

Welcome To Kylin 4 On Cloud Deployment Solution!

Apache Kylin community released Kylin 4.0 with a new architecture, which is dedicated to building a high-performance and low-cost OLAP engine. The architecture of Kylin 4.0 supports the separation of storage and computing, which enables Kylin users to run Kylin 4.0 by adopting a more flexible and elastically scalable cloud deployment method.

For the best practices of Kylin4 on the cloud, Apache Kylin community contributes a tool to deploy kylin4 clusters on AWS cloud easily and conveniently.

Introduction About This Tool

Features

  1. Deploy a Kylin4 cluster on Ec2 with Spark Standalone mode in 10 minutes.
  2. Support to scale nodes (Kylin & Spark Worker) quickly and conveniently.
  3. Improve performance for the query of Kylin4 in using Local Cache + Soft Affinity feature (Experimental Feature), please check the details.
  4. Support to monitor the status of the cluster with the Prometheus server and Granfana.

Architecture

When cluster(s) created, services and nodes will be like below:

architecture

  • Every stack module means related services will be controlled by a stack.
  • The read-write separated cluster will be easily created as same as the image of architecture above.
  • Services are created as the number order from 1 to 5.
  • Every machine node is presented by a white box.
  • Kylin Node and Spark Worker Node can be easy to scale.
  • Whole clusters will have only one RDS and only one machine node which contains Prometheus Server and Hive MetaStore service.

Quick Start

Notes

  1. For more details about cost of tool, see document cost calculation.
  2. For more details about commands of tool, see document commands.
  3. For more details about the prerequisites of tool, see document prerequisites.
  4. For more details about advanced configs of tool, see document configuration and advanced configuration.
  5. For more details about monitor services supported by tool, see document monitor.
  6. For more details about troubleshooting, see document troubleshooting.
  7. The current tool has already opened the public port for some services. You can access the service by public IP of related EC2 instances.
    1. SSH: 22.
    2. Granfana: 3000.
    3. Prometheus: 9090, 9100.
    4. Kylin: 7070.
    5. Spark: 8080, 4040.
    6. MDX for Kylin: 7080.
  8. More about cloudformation syntax, please check aws website.
  9. The current Kylin version is 4.0.1.
  10. The current Spark version is 3.1.1.