| commit | cf6bc3403e25353c51476425439b4e3ca927eae9 | [log] [tgz] |
|---|---|---|
| author | Kazuto Iris <78157415+kazutoiris@users.noreply.github.com> | Thu Oct 30 15:11:30 2025 +0800 |
| committer | GitHub <noreply@github.com> | Thu Oct 30 15:11:30 2025 +0800 |
| tree | 1b80945b3820d14290cdaeb7efdc3fbe849dd95f | |
| parent | 049ad8ed51454e2f4e1cfff581aee8a10d6246f2 [diff] |
Merge upstream changes and apply additional improvements (#5279) * feat: merge from upstream --- * Optimize task planner only create job and codeLogical Task (peacewong) * Add Job ID information to the log of the entire link of the task (peacewong) * Fix log out cookie not to clear (peacewong) * Task progress exceeding 1 will give fake progress (peacewong) * Enable to custom the storage path for yarn logs in ECM. (davidhua) * Change the default value of blank-placeholder in Flink engine; (davidhua) * optimize errorcode (peacewong) * Clean the job resources after submitting of flink-jar job in flink engine. (davidhua) * Exclude the FLINK_SHIP_DIRECTORIES from server in cleaner of Flink Engine. (davidhua) * code format (peacewong) * engineconn-plugin-core - add config for MaxMetaspaceSize (alexkun) * remove jdk8 judge (peacewong) * optimize engine log (#98) (huangKai-2323) * update task cancel msg (#97) (huangKai-2323) * update log (#94) (huangKai-2323) * fix shell upload blocking (#95) (huangKai-2323) * 【1.1.10】Fix the garbled code problem of spark function (#93) (huangKai-2323) * set version 1.1.10-wedatasphere (casionone) * 【1.1.10】Repair of the parameter validation problem configured by the management console (#96) (huangKai-2323) * fix yarn resource add + not correct (#102) (Casion) * optimize scriptis code print (#103) (Casion) * support spark.external.default.jars (#101) (Casion) * spark task add resource info log (#104) (Casion) * 【1.1.10】Entry tenant label supports common configuration (#91) (huangKai-2323) * fix csv export possible missing escape characters (#105) (Casion) * Improve exception information and add path information (#4351) Co-authored-by: casionone <casionone@gmail.com> (binbincheng) * revert of shell upload blocking (casionone) * move dir check to HDFSCacheLogWriter.init (#100) (aiceflower) * Dev 1.3.2 update tenant (#106) (huangKai-2323) * Fix custom variable settings (#107) (Casion) * engine bml resource compatible prefix v format (#108) (Casion) * Revert "move dir check to HDFSCacheLogWriter.init (#100)" (#109) (peacewong) * Fix unlock (peacewong) * Fix unlock (#111) (peacewong) * fix NPE bug (#112) (Casion) * Fix unlock (peacewong) * update to retry exception (peacewong) * remove default value 0 (peacewong) * 1.1.10 webank bugfix (#116) (Casion) * code review optimize (peacewong) * optimize about submit empty code after code preprocessing (#113) (Casion) * Fix resultset (#115) (Casion) * Dev 1.3.2 fix config cache bug (#117) (huangKai-2323) * Revert "support spark.external.default.jars (#101)" (casionone) * Dev 1.3.2 update tenant (#114) (huangKai-2323) * scala support jar (#119) (Casion) * fix(common util)：fix getStringToMap function bug about ECResourceInfoUtils (#121) (ahaoyao) * revert DoubleType NaN (casionone) * fix spark resource log NPE (casionone) * Revert "【1.1.10】Fix the garbled code problem of spark function (#93)" (casionone) * Code optimization (#125) (huangKai-2323) * bug fix and feature optimize (#124) (Yonghao Mei) * add 1.1.10 webank web (#127) (Yonghao Mei) * feat:Add single test to ByteTimeUtils and fix code that triggers null pointer exception (#126) (ahaoyao) * fix：Fix spelling errors in words (#129) (ahaoyao) * feat：add ldap-userNameFormat example value (#128) (ahaoyao) * Revert "Fix custom variable settings (#107)" This reverts commit dd10fbcc (casionone) * fix JobHistoryQueryServiceImpl cache bug (#133) (huangKai-2323) * update spark datatype (huangKai-2323) * Compatible with trino version 407. (HmhWz) * add dss gateway support parser. (#135) (mhHao) * Supports the conversion of positive and negative byte numbers to gb (ahaoyao) * Optimization of token exception scenario description (ahaoyao) * Basic Information Management Delete Interface Add Administrator Authentication (ahaoyao) * 1. Add the eclist interface input parameter: queueName 2. Add the eclist interface parameter: lastUnlockTimestamp (huangKai-2323) * adjust sqoop ec (casionone) * add datasource dependency (casionone) * add missing import (casionone) * fix spark garbled code bug (huangKai-2323) * fix spark garbled code bug (huangKai-2323) * add instanceName parameters (huangKai-2323) * Code Formatting (huangKai-2323) * adjust dss gateway support (casionone) * Code Formatting (huangKai-2323) * add doubleToStringTest (huangKai-2323) * fix Fixed session issue (peacewong) * update to ms (peacewong) * format code (huangKai-2323) * format code (huangKai-2323) * [1.1.11] Optimize UDF function logic (#131) (huangKai-2323) * update get configuration logic (#140) (aiceflower) * fix class no found (#144) (Casion) * update test (peacewong) * add log (peacewong) * Parse to specific instance for flow/entrance/kill in DSSGatewayParser. (HmhWz) * version num to 1.1.11-wedatasphere (casionone) * fix:select exclude route label is empty (#147) (Casion) * added askEngineHook (#143) (Casion) * 1.1.11 webank bdpclient (#142) (Casion) * Optimize jdbc driver (peacewong) * fix Fixed session issue (peacewong) * update to ms (peacewong) * add log (peacewong) * add license (peacewong) * Dev 1.1.11 ec list add unlock time (#148) (huangKai-2323) * update var name (peacewong) * Fix error Status call back (peacewong) * Dev 1.4.0 jdbc optimize (#149) (peacewong) * fix eclist api:filter after group by (#151) (Casion) * Optimization of token exception scenario description (#152) (ahaoyao) * ec list add unlock time---code_view (#153) (huangKai-2323) * optimize udf permissions (#150) (huangKai-2323) * Fix duplicate error codes (ahaoyao) * remove duplicate dependency of disruptor (casionone) * JDBC Optimize (peacewong) * Fix build error (peacewong) * Dev 1.1.11 optimize udf permissions (#154) (huangKai-2323) * Code Format (peacewong) * fix web bugs and optimization (#156) (Yonghao Mei) * Dev 1.1.11 webank fixoom (#157) (peacewong) * Fix issue with annotation task hang at the end of Scala code (peacewong) * add runScheduleIntercept (peacewong) * Fix storage error (peacewong) * Fix build error (peacewong) * Fix build error (peacewong) * Add creator level job limit (peacewong) * remove CacheOutputExecuteResponse (peacewong) * Fix build error (peacewong) * The dss prod label interface forwarding problem was fixed. (HmhWz) * add scheduler conf (peacewong) * feat: do not kill ec when ecm restart part-1 (#4184) (#4452) * feat: do not kill ec when ecm restart(#4192) (CharlieYan) * fix: do not clean ec resource and service info when handler ecm close event(#4184) (#4510) (CharlieYan) * update to 1.1.13-wedatasphere (casionone) * add sql connection (peacewong) * add sql connection (#168) (Casion) * Add TicketID to Node (peacewong) * Fixed ECM not cleaning up when starting EC exception (peacewong) * move File to java dir (peacewong) * Because the information needs to be pushed to createEngine, there is no need to stop here, because create will stop (peacewong) * Fix build error (peacewong) * Fix build error (peacewong) * Fix build error (peacewong) * Fix Integration Test (peacewong) * code optimize (peacewong) * Fix NPE bug #4621 (peacewong) * itsm json fix (huangKai-2323) * print real ip (ahaoyao) * Add exception logs when kill job interface (ahaoyao) * delete setStatus (ahaoyao) * properties values support the Chinese (ahaoyao) * update error code (huangKai-2323) * Optimization of incorrect 500 scenario return results (ahaoyao) * Modify comments (ahaoyao) * remove author info (ahaoyao) * Implement custom DefaultErrorAttributes (ahaoyao) * add linkis_ps_udf_user_load index (huangKai-2323) * sync 1.4.0 code (huangKai-2323) * fix spark memory count error (huangKai-2323) * Security Work Order Upgrade (ahaoyao) * udf all add description (huangKai-2323) * Code optimization (huangKai-2323) * Code optimization (huangKai-2323) * Code optimization (huangKai-2323) * add timo out task scan (peacewong) * remove node scorer service (peacewong) * Optimization of log issues, adding queue names (ahaoyao) * 【1.1.13】fix queueresources sql bug (#166) (v-kkhuang) * 【1.1.13】interface fileinfo add totalline (#165) (v-kkhuang) * 【1.1.13】remove tenant for entrance (#163) (v-kkhuang) * 【1.1.13】Adding specified parameter validation when modifying configuration files (#180) (v-kkhuang) * 【1.1.13】 Operation and maintenance optimization: Service version information reporting (#182) (v-kkhuang) * Service support merge ps-data-source-manager ps-metadataquery into ps-datasource (#185) (Casion) * remove node scorer service invoke (peacewong) * FIX conflict (peacewong) * Fix build issue (peacewong) * add mapper (casionone) * feature add config template (casionone) * remove try and warn (peacewong) * add test for TemplateConfigKey dao (casionone) * code optimize (casionone) * fix bug and code optimize (casionone) * code optimize (casionone) * queue resources and add parameter maxapps (huangKai-2323) * fix oom & shell error optimize (casionone) * 【1.1.13】queue resources and add parameter maxapps (#190) (v-kkhuang) * Adapt data source script fallback (ahaoyao) * 【1.1.13】fix licensed check error (#192) (v-kkhuang) * spotless:apply (casionone) * add sys funtion script (#187) (ahaoyao) * A single executor engine type should not create multiple executors #4713 (peacewong) * set default value to false (peacewong) * code optimization (huangKai-2323) * optimize shell error msg (casionone) * fix shell file path bug (huangKai-2323) * Support configuration template (#195) (Casion) * Fix scala dont exit issue (peacewong) * openEngineLog check with permission (casionone) * 【1.1.13】 Operation and maintenance optimization: Service version information reporting (#203) (v-kkhuang) * 【1.1.13】Adding specified parameter validation when modifying configuration files (#201) (v-kkhuang) * openEngineLog check with historyAdmin permission configuration template optimize (casionone) * revert sql result deal with \n (casionone) * rollback (ahaoyao) * code optimization (huangKai-2323) * code optimize after code review (#209) (v-kkhuang) * code optimize after code review (#206) (Casion) * fix NPE for YarnResourceRequester (#212) (v-kkhuang) * Optimization for linkis web and new features (#211) (Yonghao Mei) * fix bug get user special limit config setting (casionone) * upgrade commons-fileupload (#214) (Casion) * fixed 'job is killed by connection lost' (casionone) * code opmitization (huangKai-2323) * fix eclist heartbeat_msg (casionone) * optimize cli (peacewong) * add hive current feature (peacewong) * code format (peacewong) * Dev 1.1.13 webank web (#217) (Yonghao Mei) * [Bugfix] Task state changes should before engine state changes (#4775) (人生有如两个橘子) * add ex exit send task log (peacewong) * add appconn plugin (xlinliu) * Revert "add appconn plugin" (xlinliu) * set timeout (#4794) (人生有如两个橘子) * update null logical (peacewong) * Fix Build error (peacewong) * remove useless method (peacewong) * add ecm instance log (huangKai-2323) * When requesting EC, print linkismanager instance information (#219) (v-kkhuang) * Optimization of token exception description (#220) (ahaoyao) * EC start with app version parameter (#224) (v-kkhuang) * Spark & Hive Engine Log Ooptimization (#221) (v-kkhuang) * hive ec supports concurrency and when ec exit push task failed log (#223) (peacewong) * support cross cluster label task (#222) (lemonjuice) * upgrade to 1.1.14-wedatasphere (casionone) * 【1.1.14】New User Configuration and Configuration Management for Linkis Management Console (#225) (v-kkhuang) * fix update rule mapper (lemonjuicelove) * fix line break bug (casionone) * code format (casionone) * fix cache log write bug (#229) (Casion) * fix null value bug (peacewong) * fix resultset (ahaoyao) * fix Duplicate resource display (ahaoyao) * Fix npe (peacewong) * merge dev-1.1.13-webank-merge-1.12-webank-flink into dev-1.1.14-webank (#236) (Alexyang) * 1. linkis-engineplugin-hive - fix compile error (alexkun) * 1. linkis-engineplugin-hive - fix compile error (alexkun) * fix ClassCastException Ljava.lang.Object; cannot be cast to scala.runtime.Nothing (casionone) * 【1.1.14】fix user conf error (#237) (v-kkhuang) * 【1.1.14】update log (#238) (v-kkhuang) * Dev 1.1.14 webank jayceyang (#239) (lemonjuice) * support ec startup params with templateid (#234) (Casion) * Fix JDBC Fixed session issue (peacewong) * add template rpc response entity (casionone) * remove hooks (peacewong) * Fix queue not exit exception (peacewong) * fix Extra invisible characters (ahaoyao) * Optimize EC request linkismanager and print logs (#242) (v-kkhuang) * Fix configuration management and user configuration bugs (#241) (v-kkhuang) * fix conf template id apply api NPE (casionone) * switch on (peacewong) * 【1.1.14】code view fix (#247) (v-kkhuang) * Remove cross cluster (#246) (lemonjuice) * 【1.1.14】base conf data sql syc (#244) (v-kkhuang) * optimize token desc (ahaoyao) * Fix default null value issue (peacewong) * Fix default null value issue (peacewong) * add configKey and userConfig page; support download all result set wh… (#248) (Yonghao Mei) * add tryCatch to log and progress method (peacewong) * change LinkisErrorCodeHandler to public (#253) (Alexyang) * delete duplicate line (peacewong) * Fix thread throw exception stop (peacewong) * fix sql where label miss and (aiceflower) * Fix null instance issue (peacewong) * Fix NPE issue (peacewong) * update attempts default retry (peacewong) * update try and warn (peacewong) * Fix hive get progress npe (peacewong) * Fix build error (peacewong) * update default value (peacewong) * update null replace default false (peacewong) * service merge (huangKai-2323) * error code (huangKai-2323) * update error code dml (huangKai-2323) * update hive progress Get the completed stage for progress calculation (peacewong) * Fix fs not init bug (peacewong) * add status (huangKai-2323) * eureka instance list (#257) (v-kkhuang) * 【1.1.15】Configuration cache optimization (#255) (v-kkhuang) * update error code dml (#264) (v-kkhuang) * 【1.1.15】Spark engine add conf (#258) (v-kkhuang) * feat-4868: Delete the configurations of the master and slave nodes. (#254) (Casion) * Update vertion to 1.1.15-wedatasphere (Casion) * fix compile bug (#266) (Casion) * update bean name (#267) (aiceflower) * update lock owner field (#273) (aiceflower) * add txt file type (#270) (v-kkhuang) * 【1.1.15】fix spark conf error (#269) (v-kkhuang) * 【1.1.15】Fix Datatype Decimal Error (#265) (v-kkhuang) * 【1.1.15】Security Work Order Upgrade (#256) (v-kkhuang) * 【1.1.15】fix query config key error (#274) (v-kkhuang) * 【1.1.15】fix decimal test code (#275) (v-kkhuang) * Fix progress no push bug (peacewong) * 【1.1.15】job remove running msg (#276) (v-kkhuang) * Dev 1.1.15 webank acrossclusters (#278) (Casion) * update template conf support query with associated with the engine type (casionone) * 【1.1.15】fix spark conf bug (#279) (v-kkhuang) * 【1.1.15】filesystem rename add no admin check (#280) (v-kkhuang) * fix null exception (#282) (lemonjuice) * Dev 1.1.15 fix entrance oom (#281) (Casion) * add yarnCluster label (#284) (aiceflower) * 【1.1.15】code view fix (#286) (v-kkhuang) * [Linkis-1.1.15-Code Review] fix code review (#285) (lemonjuice) * 【1.1.15】spark code opmitization (#287) (v-kkhuang) * linkis console web optimization and add new features (#283) (Yonghao Mei) * Dev 1.1.15 webank hive (#289) (aiceflower) * 【1.1.15】bug fix (#288) (v-kkhuang) * Revert "【1.1.15】bug fix (#288)" (casionone) * bug fix ps server merege and configuration api (casionone) * optimization for console web (#290) (Yonghao Mei) * fix group name may contain with _ such as APP_A (casionone) * Fix progress get 0.1 fake progress (peacewong) * update log of match error code to error level (peacewong) * fix conf error (v_kkhuang) * switch jvm user to false (peacewong) * 【1.1.15】update dml (#292) (v-kkhuang) * update engine error msg (v_kkhuang) * update engine error msg (v_kkhuang) * update engine error msg (v_kkhuang) * update hive log4j conf (v_kkhuang) * code opmizition (v_kkhuang) * Update vertion to 1.1.16-wedatasphere (aiceflower) * update setOwner (aiceflower) * update error coode (v_kkhuang) * update error coode (v_kkhuang) * update fun name (aiceflower) * update func name (aiceflower) * update deprecated desc (aiceflower) * update error coode (v_kkhuang) * formate code (v_kkhuang) * update dml (v_kkhuang) * update msg (v_kkhuang) * add configration test (v_kkhuang) * update error msg (v_kkhuang) * Add the linkis task id to trino in the first line of tsql. (HmhWz) * code format spotless (casionone) * code support to set template conf name (casionone) * trino add query taskid info &code optimize (casionone) * update hdfs check on (aiceflower) * fix log permision (aiceflower) * update hive log4j conf (v_kkhuang) * add configration test (v_kkhuang) * code optimize and add some unit test (casionone) * add configration test (#307) (v-kkhuang) * 【1.1.16】fix regex bug (#306) (v-kkhuang) * update hive log4j conf (#308) (v-kkhuang) * linkis ecm stimate actual memory 100% (casionone) * fix result set sort bug (mayinrain) * fix bug (casionone) * fix log4j bug (v_nikeyang) * fix log4j bug (v_nikeyang) * Dev 1.1.15 webank streamis fink load yaml (#303) (yangwenzea) * fix log4j bug (v_nikeyang) * add clusterinfo and code optimize (casionone) * update error log mark red (aiceflower) * update regex (v_kkhuang) * remove tispark conf (v_kkhuang) * code opmitization (v_kkhuang) * update entrance capacity to 5000 (aiceflower) * update ecm protected to 10g by default (aiceflower) * 1. plugin-flink - fix lockservice in FlinkSpringConfiguration.scala 2. application-engineconn.yml - add param allow-bean-definition-overriding with default value true 3. code format (Alexyang) * 1. rename FlinkSpringConfiguration (Alexyang) * fix log4j bug (v_nikeyang) * fix log4j bug (v_nikeyang) * update period to 10 and fix ECM log info (aiceflower) * update value can empty (aiceflower) * fix gc log bug (v_nikeyang) * push iast fix code (v_kkhuang) * Fix linkis cli npe error (#317) (mhHao) * fix code review (v_nikeyang) * Update vertion to 1.1.17-wedatasphere (Casion) * Revert "Revert "add appconn plugin"" (xlinliu) * add appconn plugin (xlinliu) * update dss.version for linkis-engine-appconn (xlinliu) * Code optimization (v_kkhuang) * resolve The problem when application in yarn is always acceptable, the flink ec will be blocked in detach mode. (wushengyeyouya) * remove query mysql meta logic (aiceflower) * down csv ignore \n (#322) (Casion) * [1.1.17]ECM Support file download (#318) (v-kkhuang) * fix linkis_cg_lm_across_cluster_rule to linkis_ps_configuration_across_cluster_rule (#321) (lemonjuice) * [1.1.17]Linkis supports task prediction function (#319) (v-kkhuang) * add get sheets info api (#323) (aiceflower) * Dev 1.1.17 task taskprediction (#324) (v-kkhuang) * appconn code formatter (casionone) * fix some bugs and code optimizes (#325) (Casion) * fix cogfig bug (#327) (v-kkhuang) * Dev 1.1.17 task taskprediction (#326) (v-kkhuang) * fix sql check action error (casionone) * remove appconn plugin (xlinliu) * Dev 1.1.17 sheetbug (#332) (aiceflower) * [1.1.17]Add permission verification to the interface (#328) (v-kkhuang) * upgrade snakeyaml to 2.0 (#331) (aiceflower) * add cluster resource threshold (#330) (lemonjuice) * Safety work order repair (#329) (v-kkhuang) * fix bigxlsx parse error (#334) (aiceflower) * async resue engine (#335) (Casion) * wds.linkis.server.version set default value (#336) (aiceflower) * fix combile error (aiceflower) * [1.1.17] task taskprediction (#339) (v-kkhuang) * [1.1.17] add jobhistory admin (#340) (v-kkhuang) * [1.1.17]fix config bug (#338) (v-kkhuang) * fix cluster threshold logs (#337) (lemonjuice) * [1.1.17] fix hive result error (#341) (v-kkhuang) * Dev 1.1.17 task taskprediction (#343) (v-kkhuang) * Dev 1.1.17 code view fix (#344) (v-kkhuang) * parse xlsx code update (#345) (aiceflower) * add snakeyaml dep for flink (aiceflower) * modify parse excel order (aiceflower) * fix excel load into hive & log optimize (#347) (Casion) * 增加hbase引擎 (peacewong) * Modify the type of the key get from the connectParams (#349) (chenmutime) * Rollback code (#350) (v-kkhuang) * chore: add download log function; optimization for web (#348) (Yonghao Mei) * [1.1.17]application-eureka yarn to properties (#342) (v-kkhuang) * mdq revert (#352) (aiceflower) * fix scala killTask (casionone) * wds.linkis.entrance.max.capacity 5000 -> 1000 (#353) (v-kkhuang) * 回滚版本 (peacewong) * Update vertion to 1.1.18-wedatasphere (Casion) * 修改判断bug (peacewong) * Update vertion to 1.1.19-wedatasphere (Casion) * [wip][1.1.18]add monitor service (#333) (v-kkhuang) * 1. FlinkEngineConnFactory - add default ha_mode 'NONE' for manage ec (#357) (#360) (Casion) * code optimization (#359) (v-kkhuang) * Dev 1.1.18 webank acrosscluster (#358) (lemonjuice) * mss log dependency (#356) (v-kkhuang) * [wip][1.1.18]open file limit 30m (#355) (v-kkhuang) * add error code 01006 (#354) (v-kkhuang) * fix spark scala task kill (#361) (Casion) * check crossQueue length (#363) (lemonjuice) * Dev 1.1.18 webank fixbug (#366) (Casion) * 修复JDBC连接泄漏问题 (peacewong) * 优化判断逻辑 (peacewong) * 1. fix FlinkEnvConfiguration - remove dependency for HighAvailablilityOption in flink-core.jar (Alexyang) * bug fix (casionone) * fix NPE of api persistTable (casionone) * optimize spark scala job thread isInterrupted (casionone) * add across cluster queue feature in web (#371) (Yonghao Mei) * fix bug when killTask may kill main thread (casionone) * code optimize after code review (casionone) * [1.1.19]Optimize monitor code (#362) (v-kkhuang) * syn pr (#367) (v-kkhuang) * [wip][1.1.19]Optimization of Linkis Resource Prediction Interface (#364) (v-kkhuang) * update netty version 4.1.86.Final ->4.1.100.Final (#374) (v-kkhuang) * update sheetInfo result data map to array list (#375) (v-kkhuang) * [1.1.19]revent pom.xml (#376) (v-kkhuang) * optimize loop array queue cache (#378) (peacewong) * add last month variable (#379) (v-kkhuang) * fix 1.1.19 pom error (#381) (v-kkhuang) * fix if to nonEquals (peacewong) * [1.1.19] fix hive partition bug (#365) (v-kkhuang) * [1.1.19]File system openLog increases 30M limit (#370) (v-kkhuang) * [1.1.19] Tenant label adding department label management (#380) (v-kkhuang) * [1.1.19]Dev 1.1.19 add last variable (#382) (v-kkhuang) * The submit interface supports replacing the executeUser parameter (#383) (v-kkhuang) * [1.1.19] tenant add department (#385) (v-kkhuang) * 【1.1.19】 fix monitor error (#386) (v-kkhuang) * [1.1.19] fix monitor error (#388) (v-kkhuang) * optimize res path (#384) (peacewong) * [1.1.19]tenant support start/stop (#387) (v-kkhuang) * 1.add error code 01007 (#389) (v-kkhuang) * [1.1.19] mv hdfs file (#346) (v-kkhuang) * hive add application log (#368) (v-kkhuang) * [WIP] fix delete rule api (#369) (lemonjuice) * code revert (#391) (aiceflower) * fix department error (#392) (v-kkhuang) * Dev 1.1.18 mv hdfs file (#390) (v-kkhuang) * [1.1.19]fix tenant error (#393) (v-kkhuang) * update service instance (#396) (v-kkhuang) * Dev 1.1.19 add last variable (#395) (v-kkhuang) * [1.1.19] fix hdfs length error (#394) (v-kkhuang) * remove write invoke setPermission (peacewong) * fix cross rule list api (lemonjuicelove) * update case error (#397) (Casion) * Dev 1.1.19 tenant add department (#398) (v-kkhuang) * optimize ec classpath env, support task conf classpath files (peacewong) * fix cross rule query list api (lemonjuicelove) * Dev 1.1.19 hive add application log (#399) (v-kkhuang) * [1.1.19]update error code (#400) (v-kkhuang) * [1.1.19]fix chmod error (#401) (v-kkhuang) * [1.1.19]fix openFile error (#402) (v-kkhuang) * [1.1.19]fix chmod error (#403) (v-kkhuang) * [1.1.19]add archive shell (#404) (v-kkhuang) * Dev 1.1.18 fix hive partition bug (#406) (v-kkhuang) * fix python kill error (#407) (Casion) * add afterExecutorExecute (peacewong) * dev-1.1.19：across cluster code review (#408) (lemonjuice) * [1.1.19]code review fix (#409) (v-kkhuang) * fix monitor (aiceflower) * add qml conf (#411) (Casion) * update fs (aiceflower) * Update vertion to 1.1.20-wds (Casion) * remove node exist status deal (peacewong) * Fix bug oom process exit (peacewong) * code format (peacewong) * support job tags into yarn app info (peacewong) * fix slow sql (#412) (v-kkhuang) * opt code (#413) (Casion) * Feature: Add result set read column maximum limit (peacewong) * Control whether engine logs are only viewed by admin (“v_kkhuang”) * Revert "Control whether engine logs are only viewed by admin" (“v_kkhuang”) * chore: 1.1.19 web update (#410) (Yonghao Mei) * monitor fix (#415) (v-kkhuang) * update logger level warn -> error (#416) (v-kkhuang) * Optimize the download of Excel result set header color: red>black (#417) (v-kkhuang) * The number of rows in the result set cannot exceed 5000 rows (#419) (v-kkhuang) * User configuration function optimization (#418) (v-kkhuang) * Control whether engine logs are only viewed by admin (#420) (v-kkhuang) * [wip][1.1.20]ecm list add msg (#421) (v-kkhuang) * optimize concurrent ec execute logic (peacewong) * init list size reduce memory usage (peacewong) * update current ec max task num value,should < linkis.engineconn.concurrent.thread.num (peacewong) * update ec transition should to judge running task (peacewong) * [1.1.20]Jobhistory add taskids query (#414) (v-kkhuang) * get across cluster queue (#422) (aiceflower) * optimize concurrent ec execute logic (peacewong) * [wip][1.1.20]update bml select (#424) (v-kkhuang) * jdbc support params set engineversion (#425) (Casion) * Dev 1.1.20 webank acrosscluster (#426) (lemonjuice) * should throw exception (peacewong) * fix cross cluster resource bug (lemonjuicelove) * code optimize (peacewong) * update code (#427) (aiceflower) * code format (lemonjuicelove) * update isOnline (#428) (aiceflower) * Update code (Casion) * add trace log (peacewong) * fix cross cluster (lemonjuicelove) * use all path (peacewong) * add error code 13002 (#430) (v-kkhuang) * [1.1.20] update cloum length 128 ->256 (#431) (v-kkhuang) * add cross cluster notes (lemonjuicelove) * fix cross cluster resource bug (lemonjuicelove) * add across resource show switch (aiceflower) * fix entrance error (#433) (v-kkhuang) * [1.1.20]Add SDK data type return (#429) (v-kkhuang) * merge 1.1.19 webank to 1.1.20-webank (#434) (Casion) * Update acrossQueues (Casion) * remove maxnumber (Casion) * roll back (peacewong) * Revert "roll back" (peacewong) * roll back (peacewong) * remove spark tags (peacewong) * update show order (aiceflower) * Fix entrance oom add job clear method (peacewong) * code format (peacewong) * Fix entrance oom add job clear method (peacewong) * 1.Fix FS support do not close fs 2.do not to create configuration everytime (peacewong) * update version (peacewong) * Fix Test (peacewong) * add create times (peacewong) * add create times (peacewong) * Feature: Add result set read column maximum limit (peacewong) * Fix NPE (peacewong) * Fix col value length cause oom bug (peacewong) * Fix build issue (peacewong) * fix bug of spark ec not exit successfully (casionone) * add row max check (peacewong) * add row max check (peacewong) * Fix case issue (peacewong) * Fix add class path bug (peacewong) * Fix exception thrown (peacewong) * 统一结果集行列限制返回 (peacewong) * fix cache loger may missing logs (#435) (Casion) * [1.1.20] fix jobhistory bug (#441) (v-kkhuang) * [1.1.20]fix sdk data type (#440) (v-kkhuang) * fix get user-creator conf : *-* > hduser - * (#439) (v-kkhuang) * [1.1.20]fix can not update "" error (#438) (v-kkhuang) * add bml clear task sh (#437) (v-kkhuang) * Dev 1.1.20 webank acrosscluster (#442) (lemonjuice) * Fix ConcurrentEC max running number (peacewong) * need to check fileSystem == cache fileSystem (peacewong) * fix compile error (aiceflower) * 【1.1.20】buf fix (#443) (v-kkhuang) * add storage info method (peacewong) * add hook execute exception (peacewong) * 1. update task execute thread (#444) (peacewong) * Update version to 1.2.0-wds (Casion) * remove write to file log (aiceflower) * code review fix (#445) (v-kkhuang) * update param check (aiceflower) * [1.1.20]support dss crosscluster filiter (#446) (v-kkhuang) * [1.1.20]fix bml get new version error (#447) (v-kkhuang) * [1.1.20]revent row limit 5000 (#448) (v-kkhuang) * add hive ds annotation (aiceflower) * merge 1.1.20 to 1.2.0 (#449) (Casion) * update monitor logger print (#450) (v-kkhuang) * chore: optimize log component; adjust globalhistory search condition; bug fix (#451) (Yonghao Mei) * Update version to 1.3.0-wds (Casion) * update openfile api remove col length limit for data service (#453) (#455) (Casion) * 代码格式化 (peacewong) * 优化状态翻转逻辑 (peacewong) * 增加登录态缓存 (peacewong) * fix jdbc nullexpection error (#457) (v-kkhuang) * Dev 1.3.0 johnnwang (#456) (peacewong) * add taskId label (#460) (aiceflower) * add participle to the pipeline (#458) (v-kkhuang) * [1.3.0]add error code 13010, 43019 (#459) (v-kkhuang) * support appconn update healthy status (#461) (aiceflower) * [1.3.0]Task logs support Yarn link redirection (#462) (v-kkhuang) * add healthy log (#463) (aiceflower) * 为并发引擎增加execute once 自动退出 (peacewong) * Code optimization (#464) (v-kkhuang) * add shutdown for unhealthy node (#465) (aiceflower) * fix column_limit_display not set error (aiceflower) * code format (aiceflower) * add result col over size msg (aiceflower) * code format (aiceflower) * add enableLimit for client (aiceflower) * update error code 22001 (#466) (v-kkhuang) * update task log url (#467) (v-kkhuang) * Optimize log data yarn URL (#468) (v-kkhuang) * upd: ecm management page add batch option;Optimize large result set displaying;across cluster queue management input limit adjust (#469) (Yonghao Mei) * Update version to 1.4.0-wds (Casion) * [1.4.0]New department administrator added to the global history page (#473) (v-kkhuang) * JobHistoryClear adds configurable days (#472) (v-kkhuang) * Upload audit logs from Linkis to HDFS and delete them (#474) (v-kkhuang) * [1.4.0]Add ecminstance parameter filtering to the list interface (#475) (v-kkhuang) * engine reuse support resource judge (#471) (aiceflower) * remove once label to reuse ec (#470) (aiceflower) * Dev 1.3.0 webank hive fix (#477) (peacewong) * add partiton check interceptor (#478) (aiceflower) * add log (aiceflower) * filter ide creator (aiceflower) * filter ide creator (aiceflower) * code format (aiceflower) * update fs msg (#480) (v-kkhuang) * [1.4.0] department user view jobhistory log (#479) (v-kkhuang) * update reuse logic (aiceflower) * enable partition check by default (aiceflower) * remove redundent log (aiceflower) * [1.4.0]bug fix (#481) (v-kkhuang) * revert partition check (#483) (aiceflower) * Dev 1.4.0 webank appconn optimize (#482) (peacewong) * [1.4.0]bug fix (#484) (v-kkhuang) * [1.4.0]add error code 43042,43043 (#476) (v-kkhuang) * 增加db空闲异常的catch (peacewong) * fix write rs error when ThreadDath (#485) (aiceflower) * code review fix (aiceflower) * code review fix (#487) (v-kkhuang) * [1.4.0]slow sql fix (#486) (v-kkhuang) * add engineType check for ec reuse (aiceflower) * Revert "add engineType check for ec reuse" (aiceflower) * ec reuse add engine type judge (aiceflower) * add delimiter for args param (#490) (GLeXios) * [1.4.0]Monitor Code optimization (#489) (v-kkhuang) * 修复offline的异常 (peacewong) * Update version to 1.5.0-wds (Casion) * Security Work Order Repair (#491) (v-kkhuang) * 调整强退的逻辑和次数 (peacewong) * chore: 1.4.0 web (#493) (Yonghao Mei) * support jdbc ec (#503) (aiceflower) * add hive cluster task support (#496) (Casion) * [1.5.0]Obtain MD5 value based on hdfs file name (#497) (v-kkhuang) * [1.5.0]Add token verification interface (#499) (v-kkhuang) * add service version (#500) (v-kkhuang) * [1.5.0]Guidelines for adding resource parameters in task logs (#501) (v-kkhuang) * hive add mapreduce.job.reduce.slowstart.completedmaps conf (#494) (v-kkhuang) * monitor add dept task index to ims (#498) (v-kkhuang) * Optimize error code information (#506) (v-kkhuang) * [1.5.0]token add audit log (#505) (v-kkhuang) * hive cluseter support oncelabel (#507) (aiceflower) * 1.fix linkiscli npe bug. 2.support jobserver/BdpClient config (#508) (mhHao) * 1.5.0特性 (#509) (peacewong) * add once label (aiceflower) * Dev 1.5.0 logs add conf tips (#511) (v-kkhuang) * [1.5.0]Code optimization (#510) (v-kkhuang) * 增加随机延迟 (peacewong) * add once label for hive (aiceflower) * code format (#512) (aiceflower) * update add once label logic (aiceflower) * code optimization (aiceflower) * 修改引擎的task id为唯一的任务id (peacewong) * [1.5.0]Fix a bug where resources are negative (#513) (v-kkhuang) * update error msg (#514) (aiceflower) * update error message (aiceflower) * runtime label judge (aiceflower) * Deal with empty extraParams on starrocks (#515) (jefftlin) * optimization code (aiceflower) * add log (aiceflower) * add default for empty label (aiceflower) * Dev 1.5.0 bug fix (#516) (v-kkhuang) * 修改默认值为5 (peacewong) * add sso login (peacewong) * 修改方法 (peacewong) * 将sso-login开头的请求处理都单独 (peacewong) * remove log (aiceflower) * [1.5.0]code review fix (#518) (v-kkhuang) * fix hive log yarn url bug (#517) (v-kkhuang) * 增加上固定数 (peacewong) * Spring 升级 (peacewong) * fix build (peacewong) * fix build (peacewong) * fix build (peacewong) * Spring 升级 (peacewong) * fix build (peacewong) * Spring 升级支持 (peacewong) * 修复build 问题 (peacewong) * 移除knife4j (peacewong) * 移除knife4j 回退 (peacewong) * 增加依赖 (peacewong) * 排除不必要的依赖 (peacewong) * 代码优化 (peacewong) * 移除knife4j (peacewong) * 修复异常信息不打印问题 (peacewong) * 优化空指针问题 (peacewong) * hive 需要引入jersery client (peacewong) * update version to 1.6.0 (peacewong) * Add built-in variables: run_mon_now (#526) (v-kkhuang) * Modify the default expiration time of the token (#525) (v-kkhuang) * DBS interface supports returning libraries that can be written (#524) (v-kkhuang) * merge 1.5.0 to 1.6.0 (#528) (aiceflower) * ECM 支持更多退出码重试，以及移除不必要的上报 (peacewong) * 增加admin 相关信息 (peacewong) * fix file not exit error (#531) (v-kkhuang) * [1.6.0]Optimizing Linkis built-in parameters (#530) (v-kkhuang) * [1.6.0]Kill existing engine after updating configuration (#527) (v-kkhuang) * task submit to fix ec support (#529) (aiceflower) * [1.6.0]Optimize fs and use Hadoop to read (#520) (v-kkhuang) * Dev 1.6.0 kill engine by conf (#532) (v-kkhuang) * fix compile error (aiceflower) * Fix NPE (peacewong) * Security work order processing (#533) (v-kkhuang) * fix variable error (#534) (v-kkhuang) * shell 引擎设置job 信息到env id (peacewong) * 解决FS递归同名目录没有展示的问题 (peacewong) * 1.add isValid parms (#535) (v-kkhuang) * remove taskId label (#536) (aiceflower) * Fix python 进程不退出 (peacewong) * Dev 1.6.0 fix bug (#537) (v-kkhuang) * 增加日志打印 (peacewong) * chore: 1.6.0 (#538) (Yonghao Mei) * [1.6.0]spark add dependency for fix bug (#539) (v-kkhuang) * [1.6.0]code review fix (#540) (v-kkhuang) * bug fix (aiceflower) * code optimization (aiceflower) * [1.6.0] code review fix (#541) (v-kkhuang) * Fix the issue of multiple log prints (#542) (v-kkhuang) * Dev 1.6.0 fix bug (#543) (v-kkhuang) * 支持设置Feign超时参数 (peacewong) * Fix NPE (peacewong) * Once Client 优化支持单例 (peacewong) * 主动推送日志修改，支持缓存t (peacewong) * update version from 1.6.0 to 1.7.0 (aiceflower) * 日志优化 (peacewong) * 修复大小写问题 (peacewong) * EC结果集路径优化 (peacewong) * 移除无用代码 (peacewong) * Bug 修复 (peacewong) * 增加数据库相关访问的Retry (peacewong) * 修复异常设置错误问题 (peacewong) * merge 1.6.0 update to 1.7.0 (#545) (aiceflower) * update template match multline (aiceflower) * 调整pes的多个模块 (peacewong) * merge server module of basedata-manager、errorde-server、variable、scriptis-dev-server to be linkis-publicservice module (peacewong) * 修复编译问题 (peacewong) * 修复编译问题 (peacewong) * 解决编译问题 (peacewong) * Optimization of Jobhistory Task Information Query Interface (“v_kkhuang”) * revent Optimization of Jobhistory Task Information Query Interface (“v_kkhuang”) * add nebula ecp (#553) (aiceflower) * update monitor sh timeout conf && ec clear sh timeout conf (#552) (v-kkhuang) * [wip][1.7.0]jobhistory add fields such as whether to reuse engines (#548) (v-kkhuang) * Dev 1.7.0 jobhistory upadte get (#554) (v-kkhuang) * [wip][1.7.0]ECM resource reset and user resource reset (#547) (v-kkhuang) * support query specified column with get result set (#551) (aiceflower) * fix compile error (aiceflower) * fix comiple error (aiceflower) * code format (aiceflower) * update openFile (#555) (Casion) * update openFile api (#557) (Casion) * optimization code (#558) (Casion) * fix: fix token npe (#559) (taoran1250) * [1.7.0]Add download button to the history of the management console (#560) (v-kkhuang) * code optimization (#561) (Casion) * Resource reset code optimization (#564) (v-kkhuang) * fix: optimizing code (#562) (taoran1250) * Dev 1.7.0 webank fsfix (#563) (aiceflower) * [1.7.0]Dev 1.7.0 bug fix (#565) (v-kkhuang) * add code type and run type relation for nebula (aiceflower) * linkis-cli support parameter passing in http headers. (HmhWz) * 代码格式化 (peacewong) * 增加请求拦截耗时打印 (peacewong) * Dev 1.7.0 bug fix (#567) (v-kkhuang) * code optimization (aiceflower) * fix gateway start error (aiceflower) * Dev 1.7.0 bug fix (#569) (v-kkhuang) * 修改代码更加优雅 (peacewong) * Dev 1.7.0 bug fix (#570) (v-kkhuang) * fix IAST bug (#568) (taoran1250) * [1.7.0]code review fix (#571) (v-kkhuang) * Fix Security Work Order: Upgrade Spring Version:5.3.27 -> 5.3.34 (#573) (v-kkhuang) * 【1.7.0】SDK interface adds new parameters (#574) (v-kkhuang) * chore: add spring-boot-actuator dependency (#572) (taoran1250) * optimization & format code (aiceflower) * IAST BUG FIX (#575) (taoran1250) * 【1.7.0】fix jobhistory query with code (#576) (v-kkhuang) * [1.7.0]fix download slowly (#577) (v-kkhuang) * 结果集路径bug修复 (peacewong) * Dev 1.7.0 bug fix (#578) (v-kkhuang) * 代码格式化 (peacewong) * 修复ConcurrentHashMap 遍历Values的并发问题 (peacewong) * 增加用户部门信息查询的接口 (peacewong) * rpc white check bug fix (aiceflower) * 增加Spark opts (peacewong) * Dev 1.7.0 bug fix (#579) (v-kkhuang) * Dev 1.7.0 bug fix (#580) (v-kkhuang) * Dev 1.7.0 bug fix (#581) (v-kkhuang) * optimazation code (aiceflower) * code optimization (aiceflower) * null value bug fix (aiceflower) * update sdk action (#589) (v-kkhuang) * merge 1.7.1 to 1.8.0 (#592) (aiceflower) * YYYYMMddHH 变量时分秒修改为取当前值 (peacewong) * Fix jdbc 复杂类型的值转换bug (peacewong) * feat: add variables print in log && add nebula space config (#595) (taoran1250) * 1.8.0 requirements (#596) (aiceflower) * code review (#598) (aiceflower) * update python module download dir to engine tmp dir (aiceflower) * update version from 1.7.0 to 1.8.0 (aiceflower) * code format (aiceflower) * update python hook support zip file (aiceflower) * feat: add multiple result set support (#601) (finyuq) * update rsa key length (aiceflower) * feat: change ims alert to ecc_receiver (#600) (taoran1250) * code format (aiceflower) * feat: add python version to Log && mybatis-plus dependency update (#602) (taoran1250) * update python error code (aiceflower) * support nebula idel kill (aiceflower) * support jdbc unallow batch kill (aiceflower) * 增加管理员判断 (peacewong) * chroe: Source Check commit (#603) (taoran1250) * remove nebula to allow batch kill engine (aiceflower) * code review fix & token update (aiceflower) * update encryption algorithm (#604) (aiceflower) * fix:code review fix (#605) (taoran1250) * chore: 1.8.0 (#606) (Yonghao Mei) * revert crypto utils (aiceflower) * fix:spark引擎打印python版本问题修复 (#607) (taoran1250) * remove jdbc query limit (aiceflower) * code format (aiceflower) * 优化undone task 功能 (peacewong) * update default token (aiceflower) * Fix fs close 问题 (peacewong) * merge log (aiceflower) * chore: update engine configuration and reuse service (aiceflower) * 修复HDFS重复close问题 (peacewong) * 增加kill all方法调用 (peacewong) * feat: root.xxx适配 (#678) (taoran1250) * feat: Change the ENGINE_SEND_LOG_TO_ENTRANCE_LIMIT_ENABLED configuration to be obtained at runtime (#684) (taoran1250) * feat: Parse UDF function name interface submission (#685) (taoran1250) * Dev 1.10.0 webank merge udf register function (#686) (taoran1250) * fix: getudffuction 权限问题修复 (#691) (taoran1250) * fix: sonar fix (#692) (taoran1250) * fix: code review fix (#699) (taoran1250) * fix: udf get function name optimization (#704) (taoran1250) * load instance resource兼容性适配 (#709) (taoran1250) * udf bug fix (#711) (taoran1250) * entrance suport priority queue (#679) (aiceflower) * code optimization (aiceflower) * deal with double (aiceflower) * code check for merge update (aiceflower) * fix label npe (aiceflower) * fix map to json error (aiceflower) * code review fix (#700) (aiceflower) * code optimization (aiceflower) * bug fix (aiceflower) * code optimization (aiceflower) * fix log error (aiceflower) * log optimization (aiceflower) * fix comiple error (aiceflower) * fix toInt error (aiceflower) * fix kill engine bug (aiceflower) * set hdfs not close user to hadoop (aiceflower) * nebula unallow kill (aiceflower) * fix queuename contains root. (aiceflower) * fix resource error (aiceflower) * jdbc optimization (aiceflower) * spark enc udf support py, scala (#731) (aiceflower) * Dev 1.10.0 analyze job (#666) (v-kkhuang) * add aes enc (aiceflower) * Adjust the execution time of the user department information synchronization task (#674) (v-kkhuang) * Dev 1.10.0 monitor starrock update (#667) (v-kkhuang) * Dev 1.10.0 sdk support shell return (#670) (v-kkhuang) * Slow query repair (#672) (v-kkhuang) * push updaet udf kill engine (#677) (v-kkhuang) * fix datasource error (#680) (v-kkhuang) * Code Optimization (#681) (v-kkhuang) * Dev 1.10.0 bug fix (#690) (v-kkhuang) * Dev 1.10.0 bug fix (#693) (v-kkhuang) * Dev 1.10.0 bug fix (#694) (v-kkhuang) * Dev 1.10.0 bug fix (#697) (v-kkhuang) * [1.9.0]Added StarRocks task timeout alarm and timeout kill function (#615) (v-kkhuang) * [1.9.0]Data source password encryption enhancement (#612) (v-kkhuang) * [1.9.0]Add department level result set restriction configuration and inspection (#617) (v-kkhuang) * [1.9.0]Fix Hive dependency issues (#618) (v-kkhuang) * fix datasource error (#620) (v-kkhuang) * Python module optimization upload (#619) (v-kkhuang) * Dev 1.9.0 fix bug (#621) (v-kkhuang) * Dev 1.9.0 fix bug (#622) (v-kkhuang) * Dev 1.9.0 fix bug (#623) (v-kkhuang) * add aes (v-kkhuang) * bug fix (v-kkhuang) * Dev 1.9.0 bug fix (#629) (v-kkhuang) * Dev 1.9.0 bug fix (#634) (v-kkhuang) * Dev 1.9.0 bug fix (#635) (v-kkhuang) * Dev 1.9.0 bug fix (#636) (v-kkhuang) * Dev 1.9.0 bug fix (#637) (v-kkhuang) * Dev 1.9.0 bug fix (#639) (v-kkhuang) * [1.9.0]code reivew fix (#641) (v-kkhuang) * bug fix (aiceflower) * bug fix (aiceflower) * Dev 1.9.0 code view fix (#643) (v-kkhuang) * Dev 1.9.0 code view fix (#644) (v-kkhuang) * Dev 1.9.0 code view fix (#645) (v-kkhuang) * Add a method to obtain the result set, getResultSet is used to obtain the result set (#646) (v-kkhuang) * 【1.9.0】Security Work Order Fix (#647) (v-kkhuang) * Dev 1.9.0 code view fix (#648) (v-kkhuang) * Code optimization (#650) (v-kkhuang) * Dev 1.9.0 code view fix (#651) (v-kkhuang) * Dev 1.9.0 code view fix (#652) (v-kkhuang) * [1.10.0]code review fix (#701) (v-kkhuang) * Dev 1.10.0 bug fix (#703) (v-kkhuang) * Dev 1.10.0 bug fix (#705) (v-kkhuang) * Dev 1.10.0 bug fix (#708) (v-kkhuang) * Code Optimization (#712) (v-kkhuang) * sync analyze sh (#713) (v-kkhuang) * 【1.10.0】Add HDFS file inspection interface and optimize related functions (#716) (v-kkhuang) * Dev 1.10.0 check hdfs file (#721) (v-kkhuang) * Dev 1.10.0 check hdfs file (#725) (v-kkhuang) * Dev 1.10.0 check hdfs file (#726) (v-kkhuang) * Dev 1.10.0 check hdfs file (#732) (v-kkhuang) * code format (aiceflower) * fix scheuler module compile error (aiceflower) * fix linkis-mybatis module compile error (aiceflower) * fix linkis-storage module compile error (aiceflower) * fix linkis-computation-orchestrator module compile error (aiceflower) * Generate spark sql based on the jdbc datasource and optimized code (#4570) (ChengJie1053) * code merge (aiceflower) * add ranger auth support & optimizat hive metastore (aiceflower) * fix linkis-udf modul compile error (aiceflower) * engine reuse add python version judgement (aiceflower) * engine reuse add python version judegement (aiceflower) * code format (aiceflower) * merge entrance module (guoshupei) * compile error fix (aiceflower) * support spark3 (v-kkhuang) * fix json key duplicate (#737) (#740) (aiceflower) * pg sql code optimization (aiceflower) * support execute jdbc task By ip and port config (aiceflower) * upd: 资源管理后台数据结构更改 (mayinrain) * upd: tag样式优化 (mayinrain) * * License add (v-kkhuang) * fix: 编译缩进检查不通过 (mayinrain) * fix: js文件证书格式修复 (mayinrain) * code merge (aiceflower) * [1.12.0]Add udflog to the engine logs directory (#756) (v-kkhuang) * Support ai sql (#759) (v-kkhuang) * Support department ID one to many (#758) (v-kkhuang) * 【1.12.0】Optimize the reuse engine configuration rollback (#760) (v-kkhuang) * support aisql (#762) (aiceflower) * remove jdbc multi (#763) (v-kkhuang) * support aisql (#764) (aiceflower) * ai sql (#766) (aiceflower) * fix compile error (aiceflower) * code optimization (#768) (aiceflower) * update conf (aiceflower) * code optimization (#770) (aiceflower) * update python version (aiceflower) * update ldap expire time to 20m (aiceflower) * update conf (aiceflower) * code optimization (aiceflower) * update progress (aiceflower) * update code (aiceflower) * update progress (aiceflower) * code optimization (aiceflower) * code optimization (aiceflower) * aisql code merge (aiceflower) * print dynamic param (aiceflower) * code review fix (aiceflower) * code optimization (aiceflower) * code optimization (aiceflower) * code optimization (aiceflower) * code optimization (aiceflower) * code optimization (aiceflower) * update max executors (aiceflower) * Add Spark 3 default (#774) (v-kkhuang) * bug fix (#775) (v-kkhuang) * code merge (aiceflower) * add spark conf dir (#778) (v-kkhuang) * Dev 1.12.0 code review fix (#779) (v-kkhuang) * Code optimization (#786) (v-kkhuang) * 优化 Hive 和 Spark 配置重置逻辑 (#785) (v-kkhuang) * Spark supports forced conversion: user configuration, department configuration (#782) (v-kkhuang) * [1.13.0]add datasource rpc GetInfoPublishedByUserIpPortAction (#787) (v-kkhuang) * [1.13.0]Spark supports forced conversion: user configuration, department conf (#788) (v-kkhuang) * merge commonentranceparser (aiceflower) * Dev 1.13.0 webank temp (#789) (aiceflower) * code optimization (aiceflower) * code format (aiceflower) * support conn param execute task (#792) (aiceflower) * [1.13.0]Data source supports automated publishing (#791) (v-kkhuang) * merge ds code (aiceflower) * code optimization (aiceflower) * some code retry optimization (aiceflower) * Dev 1.13.0 bug fix (#793) (v-kkhuang) * code optimization (aiceflower) * [1.13.0]code opmitization (#795) (v-kkhuang) * code optimization (aiceflower) * compile error fix (aiceflower) * support error task retry (aiceflower) * template conf optimization (aiceflower) * classcastexception fix (aiceflower) * code optimization & format (aiceflower) * params optimization (aiceflower) * code optimization (aiceflower) * task error receiver sync (aiceflower) * update code to status rpc (aiceflower) * support diagnose (aiceflower) * add log (aiceflower) * bug fix (aiceflower) * code review fix (aiceflower) * code review fix (#802) (v-kkhuang) * ds optimization (aiceflower) * Dev 1.13.0 code review fix (#803) (v-kkhuang) * add conf retry switch (aiceflower) * code uptate (aiceflower) * update code add retry code (aiceflower) * Dev 1.13.0 code review fix (#805) (v-kkhuang) * fix resource reset error (#807) (v-kkhuang) * Add BML client connection information retrieval function (#809) (v-kkhuang) * support change engine type for aisql (#810) (Casion) * update template conf (aiceflower) * fix todo & adjust interceptor (#811) (Casion) * Fix resource statistics logic to correctly handle empty resources (#813) (v-kkhuang) * Dev 1.14.0 bug fix (#814) (v-kkhuang) * Dev 1.14.0 bug fix (#815) (v-kkhuang) * support rsa (Casion) * Dev 1.14.0 bug fix (#817) (v-kkhuang) * 在JobHistory的搜索和查询接口中添加了runType参数，允许根据运行类型筛选作业历史记录 (#823) (v-kkhuang) * 【1.15.0】Dev 1.15.0 sys user login (#822) (v-kkhuang) * 【1.15.0】Linkis token rsa (#821) (v-kkhuang) * spark3 dynamic params (#825) (v-kkhuang) * linkis keytab (#824) (v-kkhuang) * update conf & code format (#829) (Casion) * Code optimization (#830) (v-kkhuang) * Dev 1.15.0 code review fix (#832) (v-kkhuang) * Dev 1.15.0 webank spark3 (#836) (Casion) * add log for reuse engine (aiceflower) * add log (aiceflower) * update log (aiceflower) * update shuff (aiceflower) * Dev 1.15.0 code review fix (#837) (v-kkhuang) * The openfile interface supports masking field information (#839) (v-kkhuang) * Code optimization (#843) (v-kkhuang) * entrance and linkismanager performance optimization (#853) (Casion) * Dev 1.15.0 code review fix (#852) (v-kkhuang) * update lock timeout (aiceflower) * code merge (v-kkhuang) * code merge (aiceflower) * Dev 1.15.0 code review fix (#856) (v-kkhuang) * Dev 1.15.0 code review fix (#858) (v-kkhuang) * [1.16.0]sensetive sql check (#860) (v-kkhuang) * [1.16.0]feat: 历史任务信息获取接口添加brief参数优化 (#859) (v-kkhuang) * [1.16.0] get support engine log (#862) (v-kkhuang) * Code optimization (v-kkhuang) * engine optimization (aiceflower) * Code optimization (v-kkhuang) * optimization hive meta store (aiceflower) * sensitive check (aiceflower) * code merge (aiceflower) Co-authored-by: peacewong <wpeace1212@gmail.com> Co-authored-by: davidhua <david_hua1996@hotmail.com> Co-authored-by: alexkun <xuanyu710@qq.com> Co-authored-by: huangKai-2323 <62878639+huangKai-2323@users.noreply.github.com> Co-authored-by: casionone <casionone@gmail.com> Co-authored-by: Casion <casionone@gmail.com> Co-authored-by: binbincheng <106590848+binbincheng@users.noreply.github.com> Co-authored-by: aiceflower <kinghao007@gmail.com> Co-authored-by: peacewong <peacewong@apache.org> Co-authored-by: ahaoyao <129247228+ahaoyao@users.noreply.github.com> Co-authored-by: Yonghao Mei <73584269+mayinrain@users.noreply.github.com> Co-authored-by: HmhWz <934178858@qq.com> Co-authored-by: mhHao <934178858@qq.com> Co-authored-by: ahaoyao <v_haoyao@webank.com> Co-authored-by: CharlieYan <16526279+charlieyan24@users.noreply.github.com> Co-authored-by: ahaoyao <731753729@qq.com> Co-authored-by: v-kkhuang <62878639+v-kkhuang@users.noreply.github.com> Co-authored-by: 人生有如两个橘子 <15764973965@163.com> Co-authored-by: xlinliu <xlinliu@163.com> Co-authored-by: lemonjuice <86357693+lemonjuicelove@users.noreply.github.com> Co-authored-by: lemonjuicelove <735611140@qq.com> Co-authored-by: Alexyang <xuanyu710@qq.com> Co-authored-by: aiceflower <kinsanities@sina.com> Co-authored-by: v_kkhuang <420895376@qq.com> Co-authored-by: mayinrain <624899618@qq.com> Co-authored-by: v_nikeyang <1013195908@qq.com> Co-authored-by: yangwenzea <45863771+yangwenzea@users.noreply.github.com> Co-authored-by: wushengyeyouya <690574002@qq.com> Co-authored-by: chenmutime <chenmutime@outlook.com> Co-authored-by: “v_kkhuang” <“420895376@qq.com”> Co-authored-by: GLeXios <72007152+GLeXios@users.noreply.github.com> Co-authored-by: jefftlin <57786627+jefftlin@users.noreply.github.com> Co-authored-by: taoran1250 <543121890@qq.com> Co-authored-by: finyuq <148207754+finyuq@users.noreply.github.com> Co-authored-by: ChengJie1053 <18033291053@163.com> Co-authored-by: guoshupei <719126Liyuelynn> Co-authored-by: v-kkhuang <420895376@qq.com> * fix(gateway): fix typo in test token Fix typo in test token constant from "LINKIS-UNAVAILABLE-TOKE" to "LINKIS-UNAVAILABLE-TOKEN" * fix(format): reformat code * fix(storage): rename checkSum method to getChecksum in AzureBlobFileSystem * chore(config): update spring configuration Adjust configuration format to use `spring.config.activate.on-profile` instead of `spring.profiles` * fix(lockService): add @ConditionalOnMissingBean annotation to `lockService` * fix(scheduler): reorganize scheduler strategy constants Move `PFIFO_SCHEDULER_STRATEGY` and `FIFO_SCHEDULER_STRATEGY` constants to the top of the `SchedulerConfiguration`. * fix: resolve Java >8 compilation error in Properties.putAll usage Replace `props.putAll()` with iterative `put()` calls to avoid ambiguous method reference error between `Properties` and `Hashtable` putAll methods when using Java versions greater than 8. * feat(entrance): add progress initialization to job request Set initial progress to "0.0" when creating job request in CommonEntranceParser. This ensures that all jobs start with a defined progress value of 0.0. * fix(eureka): include jersey dependencies * chore(license): add license files for new dependencies * chore(notice): add notice files for new dependencies * ci(dlc): add bilibili.com to ignore patterns bilibili.com always returns 412 as status code. --------- Co-authored-by: peacewong <wpeace1212@gmail.com> Co-authored-by: davidhua <david_hua1996@hotmail.com> Co-authored-by: alexkun <xuanyu710@qq.com> Co-authored-by: huangKai-2323 <62878639+huangKai-2323@users.noreply.github.com> Co-authored-by: casionone <casionone@gmail.com> Co-authored-by: binbincheng <106590848+binbincheng@users.noreply.github.com> Co-authored-by: aiceflower <kinghao007@gmail.com> Co-authored-by: peacewong <peacewong@apache.org> Co-authored-by: ahaoyao <129247228+ahaoyao@users.noreply.github.com> Co-authored-by: Yonghao Mei <73584269+mayinrain@users.noreply.github.com> Co-authored-by: HmhWz <934178858@qq.com> Co-authored-by: ahaoyao <v_haoyao@webank.com> Co-authored-by: CharlieYan <16526279+charlieyan24@users.noreply.github.com> Co-authored-by: ahaoyao <731753729@qq.com> Co-authored-by: v-kkhuang <62878639+v-kkhuang@users.noreply.github.com> Co-authored-by: 人生有如两个橘子 <15764973965@163.com> Co-authored-by: xlinliu <xlinliu@163.com> Co-authored-by: lemonjuice <86357693+lemonjuicelove@users.noreply.github.com> Co-authored-by: lemonjuicelove <735611140@qq.com> Co-authored-by: aiceflower <kinsanities@sina.com> Co-authored-by: v_kkhuang <420895376@qq.com> Co-authored-by: mayinrain <624899618@qq.com> Co-authored-by: v_nikeyang <1013195908@qq.com> Co-authored-by: yangwenzea <45863771+yangwenzea@users.noreply.github.com> Co-authored-by: wushengyeyouya <690574002@qq.com> Co-authored-by: chenmutime <chenmutime@outlook.com> Co-authored-by: “v_kkhuang” <“420895376@qq.com”> Co-authored-by: GLeXios <72007152+GLeXios@users.noreply.github.com> Co-authored-by: jefftlin <57786627+jefftlin@users.noreply.github.com> Co-authored-by: taoran1250 <543121890@qq.com> Co-authored-by: finyuq <148207754+finyuq@users.noreply.github.com> Co-authored-by: ChengJie1053 <18033291053@163.com>
Linkis builds a layer of computation middleware between upper applications and underlying engines. By using standard interfaces such as REST/WS/JDBC provided by Linkis, the upper applications can easily access the underlying engines such as MySQL/Spark/Hive/Presto/Flink, etc., and achieve the intercommunication of user resources like unified variables, scripts, UDFs, functions and resource files at the same time.
As a computation middleware, Linkis provides powerful connectivity, reuse, orchestration, expansion, and governance capabilities. By decoupling the application layer and the engine layer, it simplifies the complex network call relationship, and thus reduces the overall complexity and saves the development and maintenance costs as well.
Since the first release of Linkis in 2019, it has accumulated more than 700 trial companies and 1000+ sandbox trial users, which involving diverse industries, from finance, banking, tele-communication, to manufactory, internet companies and so on. Lots of companies have already used Linkis as a unified entrance for the underlying computation and storage engines of the big data platform.
Apache Linkis | DeepWiki : https://deepwiki.com/apache/linkis
Support for diverse underlying computation storage engines : Spark, Hive, Python, Shell, Flink, JDBC, Pipeline, Sqoop, OpenLooKeng, Presto, ElasticSearch, Trino, SeaTunnel, etc.;
Support for diverse language : SparkSQL, HiveSQL, Python, Shell, Pyspark, Scala, JSON and Java;
Powerful computing governance capability : It can provide task routing, load balancing, multi-tenant, traffic control, resource control and other capabilities based on multi-level labels;
Support full stack computation/storage engine : The ability to receive, execute and manage tasks and requests for various compute and storage engines, including offline batch tasks, interactive query tasks, real-time streaming tasks and data lake tasks;
Unified context service : supports cross-user, system and computing engine to associate and manage user and system resource files (JAR, ZIP, Properties, etc.), result sets, parameter variables, functions, UDFs, etc., one setting, automatic reference everywhere;
Unified materials : provides system and user level material management, can share and flow, share materials across users, across systems;
Unified data source management : provides the ability to add, delete, check and change information of Hive, ElasticSearch, Mysql, Kafka, MongoDB and other data sources, version control, connection test, and query metadata information of corresponding data sources;
Error code capability : provides error codes and solutions for common errors of tasks, which is convenient for users to locate problems by themselves;
| Engine name | Support underlying component version (default dependency version) | Linkis Version Requirements | Included in Release Package By Default | Description |
|---|---|---|---|---|
| Spark | Apache >= 2.0.0, CDH >= 5.4.0, (default Apache Spark 3.2.1) | >=1.0.3 | Yes | Spark EngineConn, supports SQL , Scala, Pyspark and R code |
| Hive | Apache >= 1.0.0, CDH >= 5.4.0, (default Apache Hive 3.1.3) | >=1.0.3 | Yes | Hive EngineConn, supports HiveQL code |
| Python | Python >= 2.6, (default Python2*) | >=1.0.3 | Yes | Python EngineConn, supports python code |
| Shell | Bash >= 2.0 | >=1.0.3 | Yes | Shell EngineConn, supports Bash shell code |
| JDBC | MySQL >= 5.0, Hive >=1.2.1, (default Hive-jdbc 2.3.4) | >=1.0.3 | No | JDBC EngineConn, already supports ClickHouse, DB2, DM, Greenplum, kingbase, MySQL, Oracle, PostgreSQL and SQLServer, can be extended quickly Support other DB, such as SQLite |
| Flink | Flink >= 1.12.2, (default Apache Flink 1.12.2) | >=1.0.2 | No | Flink EngineConn, supports FlinkSQL code, also supports starting a new Yarn in the form of Flink Jar Application |
| Pipeline | - | >=1.0.2 | No | Pipeline EngineConn, supports file import and export |
| openLooKeng | openLooKeng >= 1.5.0, (default openLookEng 1.5.0) | >=1.1.1 | No | openLooKeng EngineConn, supports querying data virtualization engine with Sql openLooKeng |
| Sqoop | Sqoop >= 1.4.6, (default Apache Sqoop 1.4.6) | >=1.1.2 | No | Sqoop EngineConn, support data migration tool Sqoop engine |
| Presto | Presto >= 0.180 | >=1.2.0 | No | Presto EngineConn, supports Presto SQL code |
| ElasticSearch | ElasticSearch >=6.0 | >=1.2.0 | No | ElasticSearch EngineConn, supports SQL and DSL code |
| Trino | Trino >=371 | >=1.3.1 | No | Trino EngineConn， supports Trino SQL code |
| Seatunnel | Seatunnel >=2.1.2 | >=1.3.1 | No | Seatunnel EngineConn， supportt Seatunnel SQL code |
Please go to the Linkis Releases Page to download a compiled distribution or a source code package of Linkis.
For more detailed guidance see:
Note: If you want use `-Dlinkis.build.web=true` to build linkis-web image, you need to compile linkis-web first. ## compile backend ### Mac OS/Linux # 1. When compiling for the first time, execute the following command first ./mvnw -N install # 2. make the linkis distribution package # - Option 1: make the linkis distribution package only ./mvnw clean install -Dmaven.javadoc.skip=true -Dmaven.test.skip=true # - Option 2: make the linkis distribution package and docker image # - Option 2.1: image without mysql jdbc jars ./mvnw clean install -Pdocker -Dmaven.javadoc.skip=true -Dmaven.test.skip=true # - Option 2.2: image with mysql jdbc jars ./mvnw clean install -Pdocker -Dmaven.javadoc.skip=true -Dmaven.test.skip=true -Dlinkis.build.with.jdbc=true # - Option 3: linkis distribution package and docker image (included web) ./mvnw clean install -Pdocker -Dmaven.javadoc.skip=true -Dmaven.test.skip=true -Dlinkis.build.web=true # - Option 4: linkis distribution package and docker image (included web and ldh (hadoop all in one for test)) ./mvnw clean install -Pdocker -Dmaven.javadoc.skip=true -Dmaven.test.skip=true -Dlinkis.build.web=true -Dlinkis.build.ldh=true -Dlinkis.build.with.jdbc=true ### Windows mvnw.cmd -N install mvnw.cmd clean install -Dmaven.javadoc.skip=true -Dmaven.test.skip=true ## compile web cd linkis/linkis-web npm install npm run build
Due to the MySQL licensing restrictions, the MySQL Java Database Connectivity (JDBC) driver is not bundled with the official released linkis image by default. However, at current stage, linkis still relies on this library to work properly. To solve this problem, we provide a script which can help to creating a custom image with mysql jdbc from the official linkis image by yourself, the image created by this tool will be tagged as linkis:with-jdbc by default.
$> LINKIS_IMAGE=linkis:1.3.1 $> ./linkis-dist/docker/scripts/make-linkis-image-with-mysql-jdbc.sh
Please refer to Quick Deployment to do the deployment.
Linkis services could be divided into three categories: computation governance services, public enhancement services and microservice governance services
Below is the Linkis architecture diagram. You can find more detailed architecture docs in Linkis-Doc/Architecture. 
Contributions are always welcomed, we need more contributors to build Linkis together. either code, or doc, or other supports that could help the community.
For code and documentation contributions, please follow the contribution guide.
We opened an issue [Who is Using Linkis] for users to feedback and record who is using Linkis.
Since the first release of Linkis in 2019, it has accumulated more than 700 trial companies and 1000+ sandbox trial users, which involving diverse industries, from finance, banking, tele-communication, to manufactory, internet companies and so on.