Apache InLong is a one-stop integration framework for massive data that provides automatic, secure and reliable data transmission capabilities. InLong offers great power to build data analysis, modeling and other real-time applications based on streaming data.
Apache InLong recently released version 1.4.0, which closed about 364+ issues with 16+ features and 120+ optimizations. It mainly completes the real-time synchronization of the entire database to Apache Doris and the real-time synchronization of the entire database to Apache Iceberg, the standard architecture supports HTTP reporting, and the standard architecture adds MongoDB and other collection nodes. This release also completes a number of other features, including:
In version 1.4.0, Agent supports data sources such as Redis, MQTT, SQLServer, Oracle, MongoDB, etc., so that the collection capabilities of standard architecture and lightweight architecture are basically aligned and users have more choices in massive scenarios. The support of this part of the back-end capabilities is mainly completed by @iamsee123 and @haibo-duan, and the front-end part is completed by @bluewang.
In version 1.4.0, the Agent, DataProxy, and Sort modules all have indicators optimized and improved, including the reconstruction of the indicators sent by the Agent, increasing the dimension of the data Group/Stream indicators, and fixing the inaccuracy of the Prometheus Listener indicators. Thanks to @Keylchen, @pocozh, and others for their contributions.
There are many InLong service components, and there has always been a problem with high deployment thresholds. In version 1.4.0, the compatibility of docker-compose deployment is optimized, and an Apache Flink environment is built-in to help developers quickly start creating tasks. Thanks to @dockerzhang for optimizing this part.
In version 1.4.0, a lot of optimizations have been made for the heartbeat of service components, including adding data protocol when reporting, automatic registration of Agent/DataProxy component reporting, adding heartbeat management API to Manager, and optimizing multiple heartbeat status bugs. Thanks to @gosonzhang, @GanfengTan, @pocozh, @lucaspeng12138 and @haifxu for their contributions.
In version 1.4.0, InLong began to support real-time synchronization of the entire database to follow up on the needs of community users. Currently, it is the first to achieve real-time synchronization of the entire database to Doris and real-time synchronization of the entire database to Iceberg/Kafka/Doris. In the near future, the community will also realize the synchronization of the entire database. Share the details. Thanks to @thesumery, @EMsnap, @yunqingmoswu for their contributions.
For more details on the 1.4.0 release, please refer to the release notes, which detail the features, enhancements, and bug fixes for this release.
In the next version, the community will continue to add synchronization scenarios for the entire database, improve task indicators, increase system stability, and conduct stress tests on standard architectures and lightweight architectures.