InLong Changelog

Release InLong 1.12.0 - Released (as of 2024-04-21)

Agent

ISSUESummary
INLONG-9801[Umbrella][Agent] Add an agent installer module for agent installation
INLONG-9698[Improve][Agent] Optimize file collection initialization logic toIO
INLONG-9700[Improve][Agent] Optimize the message ack logic to reduce semaphore competition.
INLONG-9702[Improve][Agent] Change the data transmission interval to depend on local file configuration
INLONG-9704[Improve][Agent] Modify the default value of memory control semaphores to adapt to businesses with large amounts of data
INLONG-9710[Improve][Agent] Improve the accuracy of instance heartbeat auditing
INLONG-9712[Improve][Agent] Adjusting task configuration verification logic
INLONG-9714[Improve][Agent] SQL injection in pgjdbc
INLONG-9716[Improve][Agent] Delete useless code when storing tasks
INLONG-9721[Improve][Agent] Add a common cycle parameter to the task configuration
INLONG-9736[Improve][Agent] Make time zone a common parameter
INLONG-9772[Improve][Agent] Increase auditing for sending exceptions and resending
INLONG-9798[Improve][Agent] Add type for agent installer
INLONG-9802[Improve][Agent] Add an agent installer module for agent installation
INLONG-9806[Improve][Agent] Add installer configuration file
INLONG-9816[Improve][Agent] Add config class for installer
INLONG-9829[Improve][Agent] Add guardian scripts
INLONG-9831[Improve][Agent] Increase configuration acquisition capability
INLONG-9833[Improve][Agent] Add module state to distinguish whether the module has been downloaded or installed
INLONG-9844[Improve][Agent] Add agent installer config request class
INLONG-9848[Improve][Agent] Add the getAuthHeader function in the HttpManager class
INLONG-9850[Improve][Agent] Add a function to retrieve HttpManager in the ModuleManager class
INLONG-9852[Improve][Agent] Place the configuration item for the installation package storage path in the installation package configuration
INLONG-9858[Improve][Agent] Increase local read and write capabilities for module config
INLONG-9859[Improve][Agent] Increase installation package download capability and local MD5 computing power
INLONG-9863[Improve][Agent] To avoid data loss caused by too many supplementary files
INLONG-9889[Improve][Agent] Delete test code
INLONG-9906[Improve][Agent] Add configuration comparison logic and processing of comparison results
INLONG-9909[Improve][Agent] Add unit test for installer
INLONG-9910[Improve][Agent] Increase daily verification for process monitoring, reinstallation, or re downloading
INLONG-9922[Improve][Agent] Add a configuration copy script to copy the configuration from the installer to the agent
INLONG-9946[Improve][Agent] Verify the return code, only proceed with the subsequent process if the return code is successful
INLONG-9948[Improve][Agent] The instance class has many code similarities, abstract a common class
INLONG-9955[Improve][Agent] Rename Job to Task
INLONG-9969[Improve][Agent] Release the memory semaphore of the source only when the data is placed in the queue
INLONG-9982[Improve][Agent] Adjusting the abstraction of source code to facilitate rapid addition of sources
INLONG-9983[Improve][Agent] Renew the doc of adding sources
INLONG-9987[Improve][Agent] Fix the issue of deleting the first digit whent0
INLONG-9990[Improve][Agent] Avoid MD5 calculation functions directly return""
INLONG-9997[Improve][Agent] Handling situations where the installation package md5 remains unchanged and other parameters md5 change
INLONG-9999[Improve][Agent] Handle scenarios where the module list is empty to prevent accidental deletion
INLONG-10010[Improve][Agent] Adjust source encapsulation, keep public initialization in the base class init, and place specific source
related initialization in the subclass's initSource
INLONG-10012[Improve][Agent] Adjust task encapsulation to place common logic in the base class
INLONG-9804[Feature][Agent] Add Pulsar Source for Agent
INLONG-10006[Feature][Agent] Add MongoDB data source for Agent
INLONG-8962[Bug][Agent] Before state and after state are the same in StateCallback.call
INLONG-9683[Bug][Agent] time to remove Akka and use Apache Pekko?

Dashboard

ISSUESummary
INLONG-9708[Improve][Dashboard] Module audit query date optimization
INLONG-9723[Improve][Dashboard] Module audit id query item optimization
INLONG-9729[Improve][Dashboard] Data access sink field mapping supports underscores
INLONG-9747[Improve][Dashboard] Module audit ID query data display optimization
INLONG-9777[Improve][Dashboard] Module audit ip query display optimization
INLONG-9820[Improve][Dashboard] Update Pulsar source field
INLONG-9842[Improve][Dashboard] Data access resource details support paging
INLONG-9895[Improve][Dashboard] It also needs to support restarting when the task fails
INLONG-9963[Improve][Dashboard] Kafka and Pulsar source is missing agent ip field
INLONG-10008[Improve][Dashboard][Manager] Supplement fields of MongoDB data source for Dashboard
INLONG-9837[Bug][Dasgboard] Pulsar cluster management displays errors
INLONG-9950[Bug][Dashboard] Audit query uses incorrect end time
INLONG-10001[Bug][Dashboard] End date time initialization error on the audit page

Manager

ISSUESummary
INLONG-9689[Improve][Manager] Optimize MySQL JDBC URL check
INLONG-9706[Improve][Manager] Supports jdbc verification under multiple hosts
INLONG-9718[Improve][Manager] Transform supports function type fields
INLONG-9727[Improve][Manager] Support configuring the timezone for agent collection addresses to streamSource
INLONG-9733[Improve][Manager] Support querying audit data at the minute level
INLONG-9735[Improve][Manager] Spring Web vulnerable to Open Redirect or Server Side Request Forgery
INLONG-9752[Improve][Manager] Operation logs support querying based on successful or unsuccessful calls
INLONG-9756[Improve][Manager] flink job name should be more readable
INLONG-9768[Improve][Manager] Optimize flink job building and manage procedure
INLONG-9770[Improve][Manager] Unified compression type configuration
INLONG-9808[Improve][Manager] Set the ignoreParseError field to null
INLONG-9818[Improve][Manger] Decode Msg based on the manager's configuration
INLONG-9825[Improve][Manager] reduce the creation of RestClusterClient
INLONG-9839[Improve][Manager] Optimize the auto assign logic of SortStandalocluster
INLONG-9846[Improve][Manager] Optimize slow query SQL
INLONG-9860[Improve][Manager] Manager client supports querying workflow logs
INLONG-9873[Improve][Manager] Support adding data add tasks for file collection
INLONG-9881[Improve][Manager] Add unit test of AuditController
INLONG-9883[Improve][Manager] Add unit test of HeartbeatController
INLONG-9902[Improve][Manager] Data preview supports pulsar multi cluster
INLONG-9932[Improve][Manager] Add an agent installer module management for agent installation
INLONG-9962[Improve][Manager] Data preview supports returning header and specific field information
INLONG-9968[Improve][Manager] Support pulsar multi cluster when creating pulsar consumption groups
INLONG-9974[Improve][Manager] Data preview simplifies interface field information
INLONG-9976[Improve][Manager] Support multiple types of audit indicator queries
INLONG-9980[Improve][Manager] Remove the derby.jar file from the manager project
INLONG-9985[Improve][Manager] Support authentication params for pulsar source
INLONG-9995[Improve][Manager] Support batch saving of group information and other operations
INLONG-9884[Improve][Manager] Optimized code AuditServiceImpl.java
INLONG-9886[Improve][Manager] Optimized code DataProxyConfigRepositoryV2
INLONG-9696[Feature][Manager] Manager client support delete inlong tenant
INLONG-9773[Feature][Manager] SortSdk configuration support acquire tenant from InlongGroup
INLONG-9781[Feature][Manager] Support offline synchronization task definition
INLONG-9813[Feature][Manager] Support offline data sync management
INLONG-9822[Feature][Manager] Support flink job runtime execution mode configuration
INLONG-9862[Feature][Manager] Support submit flink job for offline data sync
INLONG-9870[Feature][Manager] Pulsar DataNode support to set compression type
INLONG-9960[Feature][Manager] Manager support to config Kafka data node
INLONG-9742[Bug][Manager] Mysql miss field data_time_zone
INLONG-9760[Bug][Manager] resource may leak due to BufferedReader not closed
INLONG-9793[Bug][Manager] Manager client workflowApi.listprocess failed to pass parameters correctly
INLONG-9827[Bug][Manager] Failed to check if the consumption group exists
INLONG-9856[Bug][Manager] Missing tenant information when listTag
INLONG-9876[Bug][Manager] manager client error message has wrong format
INLONG-9917[Bug][Manager] Manager restart data sync job failed
INLONG-9921[Bug][Mananger] manager can't stop data sync job
INLONG-9953[Bug][Manager] stop stream source failed

SDK

ISSUESummary
INLONG-9762[Bug][SDK] DataProxy SDK Connect Manager error.

Sort

ISSUESummary
INLONG-9687[Improve][Sort] Remove useless configuration items in pom.xml
INLONG-9739[Improve][Sort] onParsingBodyFailure function in FailureHandler class need add ‘head’ paramete to generate failure
metric.
INLONG-9795[Improve][Sort] Regarding the format optimization of the data type enumeration definition of the Redis connector
project module
INLONG-9913[Improve][Sort] Solve end-to-end-tests-v1.13 naming problem andtfailed.
INLONG-8948[Feature][Sort] Add Redis connector on flink 1.15
INLONG-9758[Feature][Sort] StarRocks connector support state key when initializing
INLONG-9774[Feature][Sort] Support rowdata way of sort InLong message tlog-kv format
INLONG-9788[Feature][Sort] Supports data parse that contains delimiters in kv and csv data content
INLONG-9835[Feature][Sort] Add Redis connector on flink 1.15
INLONG-9871[Feature][Sort] Use ZLIB as the default compression type of pulsar sink
INLONG-9972[Feature][Sort] Pulsar connetor should support authentication when connecting to Pulsar cluster
INLONG-9899[Feature][Sort] Pulsar Extract Node supports authentication
INLONG-10017[Feature][Sort] Definition of Sort Config
INLONG-9753[Bug][Sort] module inlong-sort-standalone's conf/sid_es_v3.conlicense
INLONG-9854[Bug][Sort] rowdata-tlogkv module artifactId is error, need change
INLONG-9875[Bug][Sort] DorisLoadNode repeat
INLONG-9879[Bug][Sort] Sort inlongmsg-rowdata-kv format parse head has error
INLONG-9925[Bug][Sort] Infinite log growth causes insufficient disk space onClickhouse)
INLONG-9965[Bug][Sort] Wrong node duration of SortStandalone Pulsar Sink
INLONG-9695[Bug][Sort] Fix kafka extract node option config building error when use upsert-kafka connector

Audit

ISSUESummary
INLONG-9807[Improve][Audit] Add debug log for audit-proxy
INLONG-10014[Improve][Audit] Add new audit SDK API for Flink Sort
INLONG-9766[Feature][Audit] Support user-defined SocketAddress loader getting AuditProxy
INLONG-9797[Feature][Audit] Audit-sdk reporting supports version number dimension
INLONG-9811[Feature][Audit] SDK supports both singleton and non-singleton usage
INLONG-9904[Feature][Audit] SDK supports checkpoint feature
INLONG-9907[Feature][Audit] Audit-service add codes of entities
INLONG-9914[Feature][Audit] Add SQL related to audit-service
INLONG-9920[Feature][Audit] Audit-service add codes of source
INLONG-9926[Feature][Audit] Audit-service support HA active and backup
INLONG-9928[Feature][Audit] Audit-service HA election through mysql
INLONG-9957[Feature][Audit] Audit-service add local cache for openapi
INLONG-9977[Feature][Audit] Audit-service increases the capabilities of openapi
INLONG-9989[Feature][Audit] Audit-store support the feature of audit-version
INLONG-10003[Feature][Audit] Audit-service supports multiple data source clusters

Other

ISSUESummary
INLONG-9934[Improve][doc] update copyright date to 2024
INLONG-9893[Improve][Common] Optimized code InLongMsg.java
INLONG-9897[Improve][Common] Code cognitive complexity modification
INLONG-9690[Bug][CI] No space left on device for build