docs/configuration/spark.md

## spark configure


| Module Name (Service Name) | Parameter Name | Default Value | Description |Used|
| -------- | -------- | ----- |----- |  -----   |
|spark|linkis.spark.yarn.cluster.jars|hdfs:///spark/cluster|spark.yarn.cluster.jars|
|spark|linkis.spark.etl.support.hudi|false|spark.etl.support.hudi|
|spark|linkis.bgservice.store.prefix|hdfs:///tmp/bdp-ide/|bgservice.store.prefix|
|spark|linkis.bgservice.store.suffix|  |bgservice.store.suffix|
|spark|wds.linkis.dolphin.decimal.precision|32 |dolphin.decimal.precision|
|spark|wds.linkis.dolphin.decimal.scale| 10 | dolphin.decimal.scale|
|spark|wds.linkis.park.extension.max.pool|2 |extension.max.pool|
|spark|wds.linkis.process.threadpool.max| 100|process.threadpool.max  |
|spark|wds.linkis.engine.spark.session.hook| | spark.session.hook|
|spark|wds.linkis.engine.spark.spark-loop.init.time| 120s |spark.spark-loop.init.time|
|spark|wds.linkis.engine.spark.language-repl.init.time| 30s| spark.language-repl.init.time |
|spark|wds.linkis.spark.sparksubmit.path| spark-submit|spark.sparksubmit.path|
|spark|wds.linkis.spark.output.line.limit|10| spark.output.line.limit|
|spark|wds.linkis.spark.useHiveContext|true| spark.useHiveContext |
|spark|wds.linkis.enginemanager.core.jar|  | enginemanager.core.jar|
|spark|wds.linkis.ecp.spark.default.jar|linkis-engineconn-core-1.2.0.jar|spark.default.jar|
|spark|wds.linkis.dws.ujes.spark.extension.timeout| 3000L |spark.extension.timeout|
|spark|wds.linkis.engine.spark.fraction.length| 30 |spark.fraction.length|
|spark|wds.linkis.show.df.max.res| |show.df.max.res|
|spark|wds.linkis.mdq.application.name| linkis-ps-datasource |mdq.application.name||
|spark|wds.linkis.dolphin.limit.len| 5000|dolphin.limit.len|
|spark|wds.linkis.spark.engine.is.viewfs.env| true | spark.engine.is.viewfs.env|
|spark|wds.linkis.spark.engineconn.fatal.log|error writing class;OutOfMemoryError|spark.engineconn.fatal.log|
|spark|wds.linkis.spark.engine.scala.replace_package_header.enable| true |spark.engine.scala.replace_package_header.enable|

Use spark yarn cluster mode,need to set label "engingeConnRuntimeMode": "yarnCluster",and need to upload the dependence of the spark to 'linkis.spark.yarn.cluster.jar'(the default value is 'hdfs:///spark/cluster')
spark dependencies include jars and configuration files，For example: '/appcom/Install/linkis/lib/linkis-engineconn-plugins/spark/dist/3.2.1/lib/*.jar','/appcom/Install/linkis/conf/*'

Precautions for using yarnCluster:
Eureka url if 127.0.0.1 should be changed to the real host, such as "127.0.0.1:20303/eureka/" should be changed to "wds001:20303/eureka/"

The spark-excel package may cause class conflicts,need to download separately,put it in spark lib
wget https://repo1.maven.org/maven2/com/crealytics/spark-excel-2.12.17-3.2.2_2.12/3.2.2_0.18.1/spark-excel-2.12.17-3.2.2_2.12-3.2.2_0.18.1.jar
cp spark-excel-2.12.17-3.2.2_2.12-3.2.2_0.18.1.jar {LINKIS_HOME}/lib/linkis-engineconn-plugins/spark/dist/3.2.1/lib

spark3 is not supported by native rocketmq-spark, and the source code needs to be modified, which can be downloaded directly from the link below
https://github.com/ChengJie1053/spark3-rocketmq-connector-jar