Transform plugin : Sql [Spark]
Use SQL to process data and support Spark's rich UDF functions
name | type | required | default value |
---|---|---|---|
sql | string | yes | - |
common-options | string | no | - |
SQL statement, the table name used in SQL is the result_table_name
configured in the Source
or Transform
plugin
Transform plugin common parameters, please refer to Transform Plugin for details
sql { sql = "select username, address from user_info", }
Use the SQL plugin for field deletion. Only the
username
andaddress
fields are reserved, and the remaining fields will be discarded.user_info
is theresult_table_name
configured by the previous plugin
sql { sql = "select substring(telephone, 0, 10) from user_info", }
Use SQL plugin for data processing, use substring functions to intercept the
telephone
field
sql { sql = "select avg(age) from user_info", table_name = "user_info" }
Use SQL plugin for data aggregation, use avg functions to perform aggregation operations on the original data set, and take out the average value of the
age
field