Google Git
Sign in
apache / inlong / refs/tags/1.5.0-RC0
tag27e43a759634f48389b1ba09d88534972b2d9e42
taggerEMsnap <zpengag@connect.ust.hk>Tue Jan 10 19:53:09 2023 +0800
object9bd8781f32a151900feb2345580b858d6a1c0229 
Tagging the 1.5.0 first Releae Candidate (Candidates start at zero)
commit9bd8781f32a151900feb2345580b858d6a1c0229[log] [tgz]
authorfuweng11 <76141879+fuweng11@users.noreply.github.com>Tue Jan 10 19:46:29 2023 +0800
committerdockerzhang <dockerzhang@apache.org>Tue Jan 10 19:47:03 2023 +0800
treef0be405594fb3447e60d6511fd636dc5aa3d991b
parent43da7e2aa1df193cb140a5ef3af217eef1cd4958 [diff]
[INLONG-7206][Manager] Fix the problem of the selectBriefList method in the InlongGroupEntity.xml file is incorrect (#7207)
2 files changed
tree: f0be405594fb3447e60d6511fd636dc5aa3d991b
  1. .github/
  2. .idea/
  3. bin/
  4. codestyle/
  5. conf/
  6. docker/
  7. inlong-agent/
  8. inlong-audit/
  9. inlong-common/
  10. inlong-dashboard/
  11. inlong-dataproxy/
  12. inlong-distribution/
  13. inlong-manager/
  14. inlong-sdk/
  15. inlong-sort/
  16. inlong-sort-standalone/
  17. inlong-tools/
  18. inlong-tubemq/
  19. licenses/
  20. .asf.yaml
  21. .gitattributes
  22. .gitignore
  23. .gitmodules
  24. .licenserc.yaml
  25. CHANGES.md
  26. CONTRIBUTING.md
  27. LICENSE
  28. NOTICE
  29. pom.xml
  30. README.md
  31. SECURITY.md
README.md

A one-stop integration framework for massive data

Build Status CodeCov Maven Central GitHub release License Twitter Slack

What is Apache InLong?

Stargazers Over TimeContributors Over Time
Stargazers over timeContributor Over Time

Apache InLong is a one-stop integration framework for massive data that provides automatic, secure and reliable data transmission capabilities. InLong supports both batch and stream data processing at the same time, which offers great power to build data analysis, modeling and other real-time applications based on streaming data.

InLong (应龙) is a divine beast in Chinese mythology who guides the river into the sea, and it is regarded as a metaphor of the InLong system for reporting data streams.

InLong was originally built at Tencent, which has served online businesses for more than 8 years, to support massive data (data scale of more than 80 trillion pieces of data per day) reporting services in big data scenarios. The entire platform has integrated 5 modules: Ingestion, Convergence, Caching, Sorting, and Management, so that the business only needs to provide data sources, data service quality, data landing clusters and data landing formats, that is, the data can be continuously pushed from the source to the target cluster, which greatly meets the data reporting service requirements in the business big data scenario.

For getting more information, please visit our project documentation at https://inlong.apache.org/. inlong-structure-en.png

Features

Apache InLong offers a variety of features:

  • Ease of Use: a SaaS-based service platform. Users can easily and quickly report, transfer, and distribute data by publishing and subscribing to data based on topics.
  • Stability & Reliability: derived from the actual online production environment. It delivers high-performance processing capabilities for 10 trillion-level data streams and highly reliable services for 100 billion-level data streams.
  • Comprehensive Features: supports various types of data access methods and can be integrated with different types of Message Queue (MQ). It also provides real-time data extract, transform, and load (ETL) and sorting capabilities based on rules. InLong also allows users to plug features to extend system capabilities.
  • Service Integration: provides unified system monitoring and alert services. It provides fine-grained metrics to facilitate data visualization. Users can view the running status of queues and topic-based data statistics in a unified data metric platform. Users can also configure the alert service based on their business requirements so that users can be alerted when errors occur.
  • Scalability: adopts a pluggable architecture that allows you to plug modules into the system based on specific protocols. Users can replace components and add features based on their business requirements.

When should I use InLong?

InLong is based on MQ and aims to provide a one-stop, practice-tested module pluggable integration framework for massive data, based on this system, users can easily build stream-based data applications. It is suitable for environments that need to quickly build a data reporting platform, as well as an ultra-large-scale data reporting environment that InLong is very suitable for, and an environment that needs to automatically sort and land the reported data.

You can use InLong in the following ways：

Supported Data Nodes (Updating)

TypeNameVersionArchitecture
Extract NodeAuto PushNoneStandard
FileNoneStandard
Kafka2.xLightweight, Standard
MongoDB>= 3.6Lightweight, Standard
MQTT>= 3.1Standard
MySQL5.6, 5.7, 8.0.xLightweight, Standard
Oracle11,12,19Lightweight
PostgreSQL9.6, 10, 11, 12Lightweight, Standard
Pulsar2.8.xLightweight
Redis2.6.xStandard
SQLServer2012, 2014, 2016, 2017, 2019Lightweight, Standard
Load NodeAuto ConsumptionNoneStandard
ClickHouse20.7+Lightweight, Standard
Elasticsearch6.x, 7.xLightweight, Standard
Greenplum4.x, 5.x, 6.xLightweight, Standard
HBase2.2.xLightweight, Standard
HDFS2.x, 3.xLightweight, Standard
Hive1.x, 2.x, 3.xLightweight, Standard
Iceberg0.12.xLightweight, Standard
Hudi0.12.xLightweight, Standard
Kafka2.xLightweight, Standard
MySQL5.6, 5.7, 8.0.xLightweight, Standard
Oracle11, 12, 19Lightweight, Standard
PostgreSQL9.6, 10, 11, 12Lightweight, Standard
SQLServer2012, 2014, 2016, 2017, 2019Lightweight, Standard
TDSQL-PostgreSQL10.17Lightweight, Standard
Doris>= 0.13Lightweight, Standard
StarRocks>= 2.0Lightweight, Standard

Build InLong

More detailed instructions can be found at Quick Start section in the documentation.

Requirements:

Compile and install:

mvn clean install -DskipTests

(Optional) Compile using docker image:

docker pull maven:3.6-openjdk-8
docker run -v `pwd`:/inlong  -w /inlong maven:3.6-openjdk-8 mvn clean install -DskipTests

after compile successfully, you could find distribution file at inlong-distribution/target.

Deploy InLong

Develop InLong

Contribute to InLong

Contact Us

  • Join Apache InLong mailing lists: | Name | Scope | | | | |:------------------------------------------------------------------------------|:--------------------------------|:----------------------------------------------------------------|:--------------------------------------------------------------------|:-----------------------------------------------------------------------------| | dev@inlong.apache.org | Development-related discussions | Subscribe | Unsubscribe | Archives |
  • Ask questions on Apache InLong Slack

Documentation

License

© Contributors Licensed under an Apache-2.0 license.