Apache InLong(incubating) is a one-stop data streaming platform that provides automatic, secure, distributed, and efficient data publishing and subscription capabilities. This platform helps you easily build stream-based data applications.
InLong was originally built at Tencent, which has served online businesses for more than 8 years, to support massive data (data scale of more than 40 trillion pieces of data per day) reporting services in big data scenarios. The entire platform has integrated 5 modules: Ingestion, Convergence, Caching, Sorting, and Management, so that the business only needs to provide data sources, data service quality, data landing clusters and data landing formats, that is, the data can be continuously pushed from the source to the target cluster, which greatly meets the data reporting service requirements in the business big data scenario.
For getting more information, please visit our project documentation at https://inlong.apache.org/en-us/ .
Apache InLong offers a variety of features:
InLong is based on MQ and aims to provide a one-stop, practice-tested module pluggable data stream access service platform，based on this system, users can easily build stream-based data applications. It is suitable for environments that need to quickly build a data reporting platform, as well as an ultra-large-scale data reporting environment that InLong is very suitable for, and an environment that needs to automatically sort and land the reported data.
InLong is only a one-stop data reporting pipeline platform. It cannot be used as a persistent data storage, nor does it support complex business logic processing on data streams.
More detailed instructions can be found at Quick Start section in the documentation.
Compile and install:
$ mvn clean install -DskipTests
(Optional) Compile using docker image:
$ docker pull maven:3.6-openjdk-8 $ docker run -v `pwd`:/inlong -w /inlong maven:3.6-openjdk-8 mvn clean install -DskipTests
after compile successfully, you could find distribution file at
InLong integrates a complete component chain for data reporting in big data scenarios, and does not support automatic installation of modules now, so we need to choose manually to install all or some modules according to actual needs. Please refer to Quick Start in our project documentation.
© Contributors Licensed under an Apache-2.0 license.