| { |
| "paragraphs": [ |
| { |
| "title": "Introduction", |
| "text": "%md\n\nThis is a tutorial for Spark SQL in PySpark (based on Spark 2.x). First we need to clarify several concepts of Spark SQL\n\n* **SparkSession** - This is the entry point of Spark SQL, you need use `SparkSession` to create DataFrame/Dataset, register UDF, query table and etc.\n* **DataFrame** - There\u0027s no Dataset in PySpark, but only DataFrame. The DataFrame of PySpark is very similar with DataFrame concept of Pandas, but is distributed. \n", |
| "user": "anonymous", |
| "dateUpdated": "2021-07-26 04:33:49.926", |
| "progress": 0, |
| "config": { |
| "tableHide": false, |
| "editorSetting": { |
| "language": "text", |
| "editOnDblClick": false, |
| "completionKey": "TAB", |
| "completionSupport": false |
| }, |
| "colWidth": 12.0, |
| "editorMode": "ace/mode/text", |
| "fontSize": 9.0, |
| "editorHide": true, |
| "results": {}, |
| "enabled": true, |
| "runOnSelectionChange": true, |
| "title": true, |
| "checkEmpty": true |
| }, |
| "settings": { |
| "params": {}, |
| "forms": {} |
| }, |
| "results": { |
| "code": "SUCCESS", |
| "msg": [ |
| { |
| "type": "HTML", |
| "data": "\u003cdiv class\u003d\"markdown-body\"\u003e\n\u003cp\u003eThis is a tutorial for Spark SQL in PySpark (based on Spark 2.x). First we need to clarify several concepts of Spark SQL\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eSparkSession\u003c/strong\u003e - This is the entry point of Spark SQL, you need use \u003ccode\u003eSparkSession\u003c/code\u003e to create DataFrame/Dataset, register UDF, query table and etc.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eDataFrame\u003c/strong\u003e - There\u0026rsquo;s no Dataset in PySpark, but only DataFrame. The DataFrame of PySpark is very similar with DataFrame concept of Pandas, but is distributed.\u003c/li\u003e\n\u003c/ul\u003e\n\n\u003c/div\u003e" |
| } |
| ] |
| }, |
| "apps": [], |
| "runtimeInfos": {}, |
| "progressUpdateIntervalMs": 500, |
| "jobName": "paragraph_1578387700849_2035279674", |
| "id": "20180530-101118_380906698", |
| "dateCreated": "2020-01-07 17:01:40.849", |
| "dateStarted": "2021-07-26 04:33:49.929", |
| "dateFinished": "2021-07-26 04:33:49.938", |
| "status": "FINISHED" |
| }, |
| { |
| "title": "Create DataFrame", |
| "text": "%md\n\nThere\u0027re 2 ways to create DataFrame\n\n* Use SparkSession to create DataFrame directly. You can either create DataFrame from RDD, List type objects and etc.\n* Use DataFrameReader to create Dataset/DataFrame from many kinds of storages that are supported by spark, such as HDFS, jdbc and etc.", |
| "user": "anonymous", |
| "dateUpdated": "2021-07-26 04:34:40.824", |
| "progress": 0, |
| "config": { |
| "tableHide": false, |
| "editorSetting": { |
| "language": "markdown", |
| "editOnDblClick": true, |
| "completionKey": "TAB", |
| "completionSupport": false |
| }, |
| "colWidth": 12.0, |
| "editorMode": "ace/mode/markdown", |
| "fontSize": 9.0, |
| "editorHide": true, |
| "title": true, |
| "results": {}, |
| "enabled": true, |
| "runOnSelectionChange": true, |
| "checkEmpty": true |
| }, |
| "settings": { |
| "params": {}, |
| "forms": {} |
| }, |
| "results": { |
| "code": "SUCCESS", |
| "msg": [ |
| { |
| "type": "HTML", |
| "data": "\u003cdiv class\u003d\"markdown-body\"\u003e\n\u003cp\u003eThere\u0026rsquo;re 2 ways to create DataFrame\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eUse SparkSession to create DataFrame directly. You can either create DataFrame from RDD, List type objects and etc.\u003c/li\u003e\n\u003cli\u003eUse DataFrameReader to create Dataset/DataFrame from many kinds of storages that are supported by spark, such as HDFS, jdbc and etc.\u003c/li\u003e\n\u003c/ul\u003e\n\n\u003c/div\u003e" |
| } |
| ] |
| }, |
| "apps": [], |
| "runtimeInfos": {}, |
| "progressUpdateIntervalMs": 500, |
| "jobName": "paragraph_1578387700850_-1934281275", |
| "id": "20180530-101515_948520659", |
| "dateCreated": "2020-01-07 17:01:40.850", |
| "dateStarted": "2021-07-26 04:34:40.824", |
| "dateFinished": "2021-07-26 04:34:40.832", |
| "status": "FINISHED" |
| }, |
| { |
| "title": "Prerequisites", |
| "text": "%md\n\n\n\n**It is strongly recommended to run the following %spark.conf paragraph first to make sure correct configuration is used.**", |
| "user": "anonymous", |
| "dateUpdated": "2021-07-26 04:33:50.131", |
| "progress": 0, |
| "config": { |
| "tableHide": false, |
| "editorSetting": { |
| "language": "markdown", |
| "editOnDblClick": true, |
| "completionKey": "TAB", |
| "completionSupport": false |
| }, |
| "colWidth": 12.0, |
| "editorMode": "ace/mode/markdown", |
| "fontSize": 9.0, |
| "editorHide": true, |
| "title": false, |
| "results": {}, |
| "enabled": true, |
| "runOnSelectionChange": true, |
| "checkEmpty": true |
| }, |
| "settings": { |
| "params": {}, |
| "forms": {} |
| }, |
| "results": { |
| "code": "SUCCESS", |
| "msg": [ |
| { |
| "type": "HTML", |
| "data": "\u003cdiv class\u003d\"markdown-body\"\u003e\n\u003cp\u003e\u003cstrong\u003eIt is strongly recommended to run the following %spark.conf paragraph first to make sure correct configuration is used.\u003c/strong\u003e\u003c/p\u003e\n\n\u003c/div\u003e" |
| } |
| ] |
| }, |
| "apps": [], |
| "runtimeInfos": {}, |
| "progressUpdateIntervalMs": 500, |
| "jobName": "paragraph_1578387700850_-1709355244", |
| "id": "20180530-110023_1756702033", |
| "dateCreated": "2020-01-07 17:01:40.850", |
| "dateStarted": "2021-07-26 04:33:50.134", |
| "dateFinished": "2021-07-26 04:33:50.142", |
| "status": "FINISHED" |
| }, |
| { |
| "title": "Spark Configuration", |
| "text": "%spark.conf\n\n# It is strongly recommended to set SPARK_HOME explicitly instead of using the embedded spark of Zeppelin. As the function of embedded spark of Zeppelin is limited and can only run in local mode.\n# SPARK_HOME \u003cyour_spark_dist_path\u003e\n\n# Uncomment the following line if you want to use yarn-cluster mode (It is recommended to use yarn-cluster mode from Zeppelin 0.8, as the driver will run on the remote host of yarn cluster which can mitigate memory pressure of zeppelin server)\n# master yarn-cluster\n\n# Uncomment the following line if you want to use yarn-client mode (It is not recommended to use yarn-client after 0.8. Because it would launch the driver in the same host of zeppelin server which will increase memory pressure of zeppelin server)\n# master yarn-client\n\n# Uncomment the following line to enable HiveContext, and also put hive-site.xml under SPARK_CONF_DIR\n# zeppelin.spark.useHiveContext true\n", |
| "user": "anonymous", |
| "dateUpdated": "2021-07-26 04:33:50.234", |
| "progress": 0, |
| "config": { |
| "editorSetting": { |
| "language": "text", |
| "editOnDblClick": false, |
| "completionKey": "TAB", |
| "completionSupport": true |
| }, |
| "colWidth": 12.0, |
| "editorMode": "ace/mode/text", |
| "fontSize": 9.0, |
| "results": {}, |
| "enabled": true, |
| "runOnSelectionChange": true, |
| "title": true, |
| "checkEmpty": true |
| }, |
| "settings": { |
| "params": {}, |
| "forms": {} |
| }, |
| "results": { |
| "code": "SUCCESS", |
| "msg": [] |
| }, |
| "apps": [], |
| "runtimeInfos": {}, |
| "progressUpdateIntervalMs": 500, |
| "jobName": "paragraph_1578387700850_-1532509261", |
| "id": "20180530-110007_162886838", |
| "dateCreated": "2020-01-07 17:01:40.850", |
| "dateStarted": "2021-07-26 04:33:50.240", |
| "dateFinished": "2021-07-26 04:33:50.246", |
| "status": "FINISHED" |
| }, |
| { |
| "title": "Create Dataset/DataFrame via SparkSession", |
| "text": "%spark.pyspark\n\n# create DataFrame from python list. It can infer schema for you.\ndf1 \u003d spark.createDataFrame([(1, \"andy\", 20, \"USA\"), (2, \"jeff\", 23, \"China\"), (3, \"james\", 18, \"USA\")]).toDF(\"id\", \"name\", \"age\", \"country\")\ndf1.printSchema()\ndf1.show()\n\n# create DataFrame from pandas dataframe\ndf2 \u003d spark.createDataFrame(df1.toPandas())\ndf2.printSchema()\ndf2.show()\n\n\n\n\n\n\n\n\n\n\n\n\n", |
| "user": "anonymous", |
| "dateUpdated": "2021-07-26 04:33:50.339", |
| "progress": 0, |
| "config": { |
| "editorSetting": { |
| "language": "python", |
| "editOnDblClick": false, |
| "completionKey": "TAB", |
| "completionSupport": true |
| }, |
| "colWidth": 6.0, |
| "editorMode": "ace/mode/python", |
| "fontSize": 9.0, |
| "title": true, |
| "results": {}, |
| "enabled": true, |
| "runOnSelectionChange": true, |
| "checkEmpty": true |
| }, |
| "settings": { |
| "params": {}, |
| "forms": {} |
| }, |
| "results": { |
| "code": "SUCCESS", |
| "msg": [ |
| { |
| "type": "TEXT", |
| "data": "root\n |-- id: long (nullable \u003d true)\n |-- name: string (nullable \u003d true)\n |-- age: long (nullable \u003d true)\n |-- country: string (nullable \u003d true)\n\n+---+-----+---+-------+\n| id| name|age|country|\n+---+-----+---+-------+\n| 1| andy| 20| USA|\n| 2| jeff| 23| China|\n| 3|james| 18| USA|\n+---+-----+---+-------+\n\nroot\n |-- id: long (nullable \u003d true)\n |-- name: string (nullable \u003d true)\n |-- age: long (nullable \u003d true)\n |-- country: string (nullable \u003d true)\n\n+---+-----+---+-------+\n| id| name|age|country|\n+---+-----+---+-------+\n| 1| andy| 20| USA|\n| 2| jeff| 23| China|\n| 3|james| 18| USA|\n+---+-----+---+-------+\n\n" |
| } |
| ] |
| }, |
| "apps": [], |
| "runtimeInfos": { |
| "jobUrl": { |
| "propertyName": "jobUrl", |
| "label": "SPARK JOB", |
| "tooltip": "View in Spark web UI", |
| "group": "spark", |
| "values": [ |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d61" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d62" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d63" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d64" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d65" |
| } |
| ], |
| "interpreterSettingId": "spark" |
| } |
| }, |
| "progressUpdateIntervalMs": 500, |
| "jobName": "paragraph_1578387700850_1345292725", |
| "id": "20180530-101750_1491737301", |
| "dateCreated": "2020-01-07 17:01:40.850", |
| "dateStarted": "2021-07-26 04:33:50.341", |
| "dateFinished": "2021-07-26 04:33:51.158", |
| "status": "FINISHED" |
| }, |
| { |
| "title": "Create DataFrame via DataFrameReader", |
| "text": "%spark.pyspark\n\nSPARK_HOME \u003d os.getenv(\u0027SPARK_HOME\u0027)\n\n# Read data from json file\n# link for this people.json (https://github.com/apache/spark/blob/master/examples/src/main/resources/people.json)\n# Use hdfs path if you are using hdfs\ndf1 \u003d spark.read.json(\"file://\" + SPARK_HOME + \"/examples/src/main/resources/people.json\")\ndf1.printSchema()\ndf1.show()\n\n# Read data from csv file. You can customize it via spark.read.options. E.g. In the following example, we customize the sep and header\ndf2 \u003d spark.read.options(sep\u003d\";\", header\u003dTrue).csv(\"file://\" + SPARK_HOME + \"/examples/src/main/resources/people.csv\")\ndf2.printSchema()\ndf2.show()\n\n# Specify schema for your csv file\nfrom pyspark.sql.types import StructType, StringType, IntegerType\n\nschema \u003d StructType().add(\"name\", StringType(), True) \\\n .add(\"age\", IntegerType(), True) \\\n .add(\"job\", StringType(), True)\n \ndf3 \u003d spark.read.options(sep\u003d\";\", header\u003dTrue) \\\n .schema(schema) \\\n .csv(\"file://\" + SPARK_HOME + \"/examples/src/main/resources/people.csv\") \ndf3.printSchema()\ndf3.show()\n\n", |
| "user": "anonymous", |
| "dateUpdated": "2021-07-26 04:33:51.241", |
| "progress": 100, |
| "config": { |
| "lineNumbers": false, |
| "editorSetting": { |
| "language": "python", |
| "editOnDblClick": false, |
| "completionKey": "TAB", |
| "completionSupport": true |
| }, |
| "colWidth": 6.0, |
| "editorMode": "ace/mode/python", |
| "fontSize": 9.0, |
| "title": true, |
| "results": {}, |
| "enabled": true, |
| "runOnSelectionChange": true, |
| "checkEmpty": true |
| }, |
| "settings": { |
| "params": {}, |
| "forms": {} |
| }, |
| "results": { |
| "code": "SUCCESS", |
| "msg": [ |
| { |
| "type": "TEXT", |
| "data": "root\n |-- age: long (nullable \u003d true)\n |-- name: string (nullable \u003d true)\n\n+----+-------+\n| age| name|\n+----+-------+\n|null|Michael|\n| 30| Andy|\n| 19| Justin|\n+----+-------+\n\nroot\n |-- name: string (nullable \u003d true)\n |-- age: string (nullable \u003d true)\n |-- job: string (nullable \u003d true)\n\n+-----+---+---------+\n| name|age| job|\n+-----+---+---------+\n|Jorge| 30|Developer|\n| Bob| 32|Developer|\n+-----+---+---------+\n\nroot\n |-- name: string (nullable \u003d true)\n |-- age: integer (nullable \u003d true)\n |-- job: string (nullable \u003d true)\n\n+-----+---+---------+\n| name|age| job|\n+-----+---+---------+\n|Jorge| 30|Developer|\n| Bob| 32|Developer|\n+-----+---+---------+\n\n" |
| } |
| ] |
| }, |
| "apps": [], |
| "runtimeInfos": { |
| "jobUrl": { |
| "propertyName": "jobUrl", |
| "label": "SPARK JOB", |
| "tooltip": "View in Spark web UI", |
| "group": "spark", |
| "values": [ |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d66" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d67" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d68" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d69" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d70" |
| } |
| ], |
| "interpreterSettingId": "spark" |
| } |
| }, |
| "progressUpdateIntervalMs": 500, |
| "jobName": "paragraph_1578387700850_581443636", |
| "id": "20180530-101930_1495479697", |
| "dateCreated": "2020-01-07 17:01:40.850", |
| "dateStarted": "2021-07-26 04:33:51.244", |
| "dateFinished": "2021-07-26 04:33:52.120", |
| "status": "FINISHED" |
| }, |
| { |
| "title": "Add New Column", |
| "text": "%spark.pyspark\n\n# withColumn could be used to add new Column\ndf1 \u003d spark.createDataFrame([(1, \"andy\", 20, \"USA\"), (2, \"jeff\", 23, \"China\"), (3, \"james\", 18, \"USA\")]).toDF(\"id\", \"name\", \"age\", \"country\")\n\ndf2 \u003d df1.withColumn(\"age2\", df1[\"age\"] + 1)\ndf2.show()\n\n# the new column could replace the existing the column if the new column name is the same as the old column\ndf3 \u003d df1.withColumn(\"age\", df1[\"age\"] + 1)\ndf3.show()\n\n# Besides using expression to create new column, you could also use udf to create new column\n# Use F.upper instead of upper, because the builtin udf of spark may conclifct with that of python, such as max\nimport pyspark.sql.functions as F\ndf4 \u003d df1.withColumn(\"name\", F.upper(df1[\"name\"]))\ndf4.show()", |
| "user": "anonymous", |
| "dateUpdated": "2021-07-26 04:33:52.144", |
| "progress": 0, |
| "config": { |
| "editorSetting": { |
| "language": "python", |
| "editOnDblClick": false, |
| "completionKey": "TAB", |
| "completionSupport": true |
| }, |
| "colWidth": 6.0, |
| "editorMode": "ace/mode/python", |
| "fontSize": 9.0, |
| "title": true, |
| "results": {}, |
| "enabled": true, |
| "runOnSelectionChange": true, |
| "checkEmpty": true |
| }, |
| "settings": { |
| "params": {}, |
| "forms": {} |
| }, |
| "results": { |
| "code": "SUCCESS", |
| "msg": [ |
| { |
| "type": "TEXT", |
| "data": "+---+-----+---+-------+----+\n| id| name|age|country|age2|\n+---+-----+---+-------+----+\n| 1| andy| 20| USA| 21|\n| 2| jeff| 23| China| 24|\n| 3|james| 18| USA| 19|\n+---+-----+---+-------+----+\n\n+---+-----+---+-------+\n| id| name|age|country|\n+---+-----+---+-------+\n| 1| andy| 21| USA|\n| 2| jeff| 24| China|\n| 3|james| 19| USA|\n+---+-----+---+-------+\n\n+---+-----+---+-------+\n| id| name|age|country|\n+---+-----+---+-------+\n| 1| ANDY| 20| USA|\n| 2| JEFF| 23| China|\n| 3|JAMES| 18| USA|\n+---+-----+---+-------+\n\n" |
| } |
| ] |
| }, |
| "apps": [], |
| "runtimeInfos": { |
| "jobUrl": { |
| "propertyName": "jobUrl", |
| "label": "SPARK JOB", |
| "tooltip": "View in Spark web UI", |
| "group": "spark", |
| "values": [ |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d71" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d72" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d73" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d74" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d75" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d76" |
| } |
| ], |
| "interpreterSettingId": "spark" |
| } |
| }, |
| "progressUpdateIntervalMs": 500, |
| "jobName": "paragraph_1578387700850_-775755394", |
| "id": "20180530-105113_693855403", |
| "dateCreated": "2020-01-07 17:01:40.850", |
| "dateStarted": "2021-07-26 04:33:52.146", |
| "dateFinished": "2021-07-26 04:33:52.914", |
| "status": "FINISHED" |
| }, |
| { |
| "title": "Remove Column", |
| "text": "%spark.pyspark\n\n\ndf1 \u003d spark.createDataFrame([(1, \"andy\", 20, \"USA\"), (2, \"jeff\", 23, \"China\"), (3, \"james\", 18, \"USA\")]).toDF(\"id\", \"name\", \"age\", \"country\")\n# drop could be used to remove Column\ndf2 \u003d df1.drop(\"id\")\ndf2.show()\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n", |
| "user": "anonymous", |
| "dateUpdated": "2021-07-26 04:33:52.947", |
| "progress": 0, |
| "config": { |
| "editorSetting": { |
| "language": "python", |
| "editOnDblClick": false, |
| "completionKey": "TAB", |
| "completionSupport": true |
| }, |
| "colWidth": 6.0, |
| "editorMode": "ace/mode/python", |
| "fontSize": 9.0, |
| "title": true, |
| "results": {}, |
| "enabled": true, |
| "runOnSelectionChange": true, |
| "checkEmpty": true |
| }, |
| "settings": { |
| "params": {}, |
| "forms": {} |
| }, |
| "results": { |
| "code": "SUCCESS", |
| "msg": [ |
| { |
| "type": "TEXT", |
| "data": "+-----+---+-------+\n| name|age|country|\n+-----+---+-------+\n| andy| 20| USA|\n| jeff| 23| China|\n|james| 18| USA|\n+-----+---+-------+\n\n" |
| } |
| ] |
| }, |
| "apps": [], |
| "runtimeInfos": { |
| "jobUrl": { |
| "propertyName": "jobUrl", |
| "label": "SPARK JOB", |
| "tooltip": "View in Spark web UI", |
| "group": "spark", |
| "values": [ |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d77" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d78" |
| } |
| ], |
| "interpreterSettingId": "spark" |
| } |
| }, |
| "progressUpdateIntervalMs": 500, |
| "jobName": "paragraph_1578387700850_-886487025", |
| "id": "20180530-112045_1274721210", |
| "dateCreated": "2020-01-07 17:01:40.850", |
| "dateStarted": "2021-07-26 04:33:52.949", |
| "dateFinished": "2021-07-26 04:33:53.319", |
| "status": "FINISHED" |
| }, |
| { |
| "title": "Select Subset of Columns", |
| "text": "%spark.pyspark\n\ndf1 \u003d spark.createDataFrame([(1, \"andy\", 20, \"USA\"), (2, \"jeff\", 23, \"China\"), (3, \"james\", 18, \"USA\")]).toDF(\"id\", \"name\", \"age\", \"country\")\n# select can accept a list of string of the column names\ndf2 \u003d df1.select(\"id\", \"name\")\ndf2.show()\n\n# select can also accept a list of Column. You can create column via $ or udf\nimport pyspark.sql.functions as F\ndf3 \u003d df1.select(df1[\"id\"], F.upper(df1[\"name\"]), df1[\"age\"] + 1)\ndf3.show()\n\n\n", |
| "user": "anonymous", |
| "dateUpdated": "2021-07-26 04:33:53.349", |
| "progress": 0, |
| "config": { |
| "editorSetting": { |
| "language": "python", |
| "editOnDblClick": false, |
| "completionKey": "TAB", |
| "completionSupport": true |
| }, |
| "colWidth": 6.0, |
| "editorMode": "ace/mode/python", |
| "fontSize": 9.0, |
| "title": true, |
| "results": {}, |
| "enabled": true, |
| "runOnSelectionChange": true, |
| "checkEmpty": true |
| }, |
| "settings": { |
| "params": {}, |
| "forms": {} |
| }, |
| "results": { |
| "code": "SUCCESS", |
| "msg": [ |
| { |
| "type": "TEXT", |
| "data": "+---+-----+\n| id| name|\n+---+-----+\n| 1| andy|\n| 2| jeff|\n| 3|james|\n+---+-----+\n\n+---+-----------+---------+\n| id|upper(name)|(age + 1)|\n+---+-----------+---------+\n| 1| ANDY| 21|\n| 2| JEFF| 24|\n| 3| JAMES| 19|\n+---+-----------+---------+\n\n" |
| } |
| ] |
| }, |
| "apps": [], |
| "runtimeInfos": { |
| "jobUrl": { |
| "propertyName": "jobUrl", |
| "label": "SPARK JOB", |
| "tooltip": "View in Spark web UI", |
| "group": "spark", |
| "values": [ |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d79" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d80" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d81" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d82" |
| } |
| ], |
| "interpreterSettingId": "spark" |
| } |
| }, |
| "progressUpdateIntervalMs": 500, |
| "jobName": "paragraph_1578387700850_2124268380", |
| "id": "20180530-113042_1154914545", |
| "dateCreated": "2020-01-07 17:01:40.850", |
| "dateStarted": "2021-07-26 04:33:53.352", |
| "dateFinished": "2021-07-26 04:33:53.931", |
| "status": "FINISHED" |
| }, |
| { |
| "title": "Filter Rows", |
| "text": "%spark.pyspark\n\ndf1 \u003d spark.createDataFrame([(1, \"andy\", 20, \"USA\"), (2, \"jeff\", 23, \"China\"), (3, \"james\", 18, \"USA\")]).toDF(\"id\", \"name\", \"age\", \"country\")\n\n# filter accept a Column \ndf2 \u003d df1.filter(df1[\"age\"] \u003e\u003d 20)\ndf2.show()\n\n# To be noticed, you need to use \"\u0026\" instead of \"\u0026\u0026\" or \"AND\" \ndf3 \u003d df1.filter((df1[\"age\"] \u003e\u003d 20) \u0026 (df1[\"country\"] \u003d\u003d \"China\"))\ndf3.show()\n\n\n\n\n\n\n", |
| "user": "anonymous", |
| "dateUpdated": "2021-07-26 04:33:53.952", |
| "progress": 0, |
| "config": { |
| "editorSetting": { |
| "language": "python", |
| "editOnDblClick": false, |
| "completionKey": "TAB", |
| "completionSupport": true |
| }, |
| "colWidth": 6.0, |
| "editorMode": "ace/mode/python", |
| "fontSize": 9.0, |
| "title": true, |
| "results": {}, |
| "enabled": true, |
| "runOnSelectionChange": true, |
| "checkEmpty": true |
| }, |
| "settings": { |
| "params": {}, |
| "forms": {} |
| }, |
| "results": { |
| "code": "SUCCESS", |
| "msg": [ |
| { |
| "type": "TEXT", |
| "data": "+---+----+---+-------+\n| id|name|age|country|\n+---+----+---+-------+\n| 1|andy| 20| USA|\n| 2|jeff| 23| China|\n+---+----+---+-------+\n\n+---+----+---+-------+\n| id|name|age|country|\n+---+----+---+-------+\n| 2|jeff| 23| China|\n+---+----+---+-------+\n\n" |
| } |
| ] |
| }, |
| "apps": [], |
| "runtimeInfos": { |
| "jobUrl": { |
| "propertyName": "jobUrl", |
| "label": "SPARK JOB", |
| "tooltip": "View in Spark web UI", |
| "group": "spark", |
| "values": [ |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d83" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d84" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d85" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d86" |
| } |
| ], |
| "interpreterSettingId": "spark" |
| } |
| }, |
| "progressUpdateIntervalMs": 500, |
| "jobName": "paragraph_1578387700850_1501705200", |
| "id": "20180530-113407_58454283", |
| "dateCreated": "2020-01-07 17:01:40.850", |
| "dateStarted": "2021-07-26 04:33:53.954", |
| "dateFinished": "2021-07-26 04:33:54.575", |
| "status": "FINISHED" |
| }, |
| { |
| "title": "Create UDF", |
| "text": "%spark.pyspark\n\ndf1 \u003d spark.createDataFrame([(1, \"andy\", 20, \"USA\"), (2, \"jeff\", 23, \"China\"), (3, \"james\", 18, \"USA\")]) \\\n .toDF(\"id\", \"name\", \"age\", \"country\")\n\n# Create udf create python lambda\nfrom pyspark.sql.functions import udf\nudf1 \u003d udf(lambda e: e.upper())\ndf2 \u003d df1.select(udf1(df1[\"name\"]))\ndf2.show()\n\n# UDF could also be used in filter, in this case the return type must be Boolean\n# We can also use annotation to create udf\nfrom pyspark.sql.types import *\n@udf(returnType\u003dBooleanType())\ndef udf2(e):\n if e \u003e\u003d 20:\n return True;\n else:\n return False\n\ndf3 \u003d df1.filter(udf2(df1[\"age\"]))\ndf3.show()\n\n# UDF could also accept more than 1 argument.\nudf3 \u003d udf(lambda e1, e2: e1 + \"_\" + e2)\ndf4 \u003d df1.select(udf3(df1[\"name\"], df1[\"country\"]).alias(\"name_country\"))\ndf4.show()\n", |
| "user": "anonymous", |
| "dateUpdated": "2021-07-26 04:33:54.655", |
| "progress": 0, |
| "config": { |
| "editorSetting": { |
| "language": "python", |
| "editOnDblClick": false, |
| "completionKey": "TAB", |
| "completionSupport": true |
| }, |
| "colWidth": 6.0, |
| "editorMode": "ace/mode/python", |
| "fontSize": 9.0, |
| "title": true, |
| "results": {}, |
| "enabled": true, |
| "runOnSelectionChange": true, |
| "checkEmpty": true |
| }, |
| "settings": { |
| "params": {}, |
| "forms": {} |
| }, |
| "results": { |
| "code": "SUCCESS", |
| "msg": [ |
| { |
| "type": "TEXT", |
| "data": "+--------------+\n|\u003clambda\u003e(name)|\n+--------------+\n| ANDY|\n| JEFF|\n| JAMES|\n+--------------+\n\n+---+----+---+-------+\n| id|name|age|country|\n+---+----+---+-------+\n| 1|andy| 20| USA|\n| 2|jeff| 23| China|\n+---+----+---+-------+\n\n+------------+\n|name_country|\n+------------+\n| andy_USA|\n| jeff_China|\n| james_USA|\n+------------+\n\n" |
| } |
| ] |
| }, |
| "apps": [], |
| "runtimeInfos": { |
| "jobUrl": { |
| "propertyName": "jobUrl", |
| "label": "SPARK JOB", |
| "tooltip": "View in Spark web UI", |
| "group": "spark", |
| "values": [ |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d87" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d88" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d89" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d90" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d91" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d92" |
| } |
| ], |
| "interpreterSettingId": "spark" |
| } |
| }, |
| "progressUpdateIntervalMs": 500, |
| "jobName": "paragraph_1578387700850_574730063", |
| "id": "20180530-113720_1986531680", |
| "dateCreated": "2020-01-07 17:01:40.850", |
| "dateStarted": "2021-07-26 04:33:54.658", |
| "dateFinished": "2021-07-26 04:33:55.479", |
| "status": "FINISHED" |
| }, |
| { |
| "title": "GroupBy", |
| "text": "%spark.pyspark\n\ndf1 \u003d spark.createDataFrame([(1, \"andy\", 20, \"USA\"), (2, \"jeff\", 23, \"China\"), (3, \"james\", 18, \"USA\")]) \\\n .toDF(\"id\", \"name\", \"age\", \"country\")\n\n# You can call agg function after groupBy directly, such as count/min/max/avg/sum\ndf2 \u003d df1.groupBy(\"country\").count()\ndf2.show()\n\n# Pass a Map if you want to do multiple aggregation\ndf3 \u003d df1.groupBy(\"country\").agg({\"age\": \"avg\", \"id\": \"count\"})\ndf3.show()\n\nimport pyspark.sql.functions as F\n# Or you can pass a list of agg function\ndf4 \u003d df1.groupBy(\"country\").agg(F.avg(df1[\"age\"]).alias(\"avg_age\"), F.count(df1[\"id\"]).alias(\"count\"))\ndf4.show()\n\n# You can not pass Map if you want to do multiple aggregation on the same column as the key of Map should be unique. So in this case\n# you have to pass a list of agg functions\ndf5 \u003d df1.groupBy(\"country\").agg(F.avg(df1[\"age\"]).alias(\"avg_age\"), F.max(df1[\"age\"]).alias(\"max_age\"))\ndf5.show()\n\n\n\n\n\n\n", |
| "user": "anonymous", |
| "dateUpdated": "2021-07-26 04:33:55.558", |
| "progress": 0, |
| "config": { |
| "editorSetting": { |
| "language": "python", |
| "editOnDblClick": false, |
| "completionKey": "TAB", |
| "completionSupport": true |
| }, |
| "colWidth": 6.0, |
| "editorMode": "ace/mode/python", |
| "fontSize": 9.0, |
| "title": true, |
| "results": {}, |
| "enabled": true, |
| "runOnSelectionChange": true, |
| "checkEmpty": true |
| }, |
| "settings": { |
| "params": {}, |
| "forms": {} |
| }, |
| "results": { |
| "code": "SUCCESS", |
| "msg": [ |
| { |
| "type": "TEXT", |
| "data": "+-------+-----+\n|country|count|\n+-------+-----+\n| China| 1|\n| USA| 2|\n+-------+-----+\n\n+-------+---------+--------+\n|country|count(id)|avg(age)|\n+-------+---------+--------+\n| China| 1| 23.0|\n| USA| 2| 19.0|\n+-------+---------+--------+\n\n+-------+-------+-----+\n|country|avg_age|count|\n+-------+-------+-----+\n| China| 23.0| 1|\n| USA| 19.0| 2|\n+-------+-------+-----+\n\n+-------+-------+-------+\n|country|avg_age|max_age|\n+-------+-------+-------+\n| China| 23.0| 23|\n| USA| 19.0| 20|\n+-------+-------+-------+\n\n" |
| } |
| ] |
| }, |
| "apps": [], |
| "runtimeInfos": { |
| "jobUrl": { |
| "propertyName": "jobUrl", |
| "label": "SPARK JOB", |
| "tooltip": "View in Spark web UI", |
| "group": "spark", |
| "values": [ |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d93" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d94" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d95" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d96" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d97" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d98" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d99" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d100" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d101" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d102" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d103" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d104" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d105" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d106" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d107" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d108" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d109" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d110" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d111" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d112" |
| } |
| ], |
| "interpreterSettingId": "spark" |
| } |
| }, |
| "progressUpdateIntervalMs": 500, |
| "jobName": "paragraph_1578387700850_1233271138", |
| "id": "20180530-114404_2076888937", |
| "dateCreated": "2020-01-07 17:01:40.850", |
| "dateStarted": "2021-07-26 04:33:55.560", |
| "dateFinished": "2021-07-26 04:33:59.547", |
| "status": "FINISHED" |
| }, |
| { |
| "title": "Join on Single Field", |
| "text": "%spark.pyspark\n\ndf1 \u003d spark.createDataFrame([(1, \"andy\", 20, 1), (2, \"jeff\", 23, 2), (3, \"james\", 18, 3)]).toDF(\"id\", \"name\", \"age\", \"c_id\")\ndf1.show()\n\ndf2 \u003d spark.createDataFrame([(1, \"USA\"), (2, \"China\")]).toDF(\"c_id\", \"c_name\")\ndf2.show()\n\n# You can just specify the key name if join on the same key\ndf3 \u003d df1.join(df2, \"c_id\")\ndf3.show()\n\n# Or you can specify the join condition expclitly in case the key is different between tables\ndf4 \u003d df1.join(df2, df1[\"c_id\"] \u003d\u003d df2[\"c_id\"])\ndf4.show()\n\n# You can specify the join type afte the join condition, by default it is inner join\ndf5 \u003d df1.join(df2, df1[\"c_id\"] \u003d\u003d df2[\"c_id\"], \"left_outer\")\ndf5.show()", |
| "user": "anonymous", |
| "dateUpdated": "2021-07-26 04:33:59.563", |
| "progress": 25, |
| "config": { |
| "editorSetting": { |
| "language": "python", |
| "editOnDblClick": false, |
| "completionKey": "TAB", |
| "completionSupport": true |
| }, |
| "colWidth": 6.0, |
| "editorMode": "ace/mode/python", |
| "fontSize": 9.0, |
| "title": true, |
| "results": {}, |
| "enabled": true, |
| "runOnSelectionChange": true, |
| "checkEmpty": true |
| }, |
| "settings": { |
| "params": {}, |
| "forms": {} |
| }, |
| "results": { |
| "code": "SUCCESS", |
| "msg": [ |
| { |
| "type": "TEXT", |
| "data": "+---+-----+---+----+\n| id| name|age|c_id|\n+---+-----+---+----+\n| 1| andy| 20| 1|\n| 2| jeff| 23| 2|\n| 3|james| 18| 3|\n+---+-----+---+----+\n\n+----+------+\n|c_id|c_name|\n+----+------+\n| 1| USA|\n| 2| China|\n+----+------+\n\n+----+---+----+---+------+\n|c_id| id|name|age|c_name|\n+----+---+----+---+------+\n| 1| 1|andy| 20| USA|\n| 2| 2|jeff| 23| China|\n+----+---+----+---+------+\n\n+---+----+---+----+----+------+\n| id|name|age|c_id|c_id|c_name|\n+---+----+---+----+----+------+\n| 1|andy| 20| 1| 1| USA|\n| 2|jeff| 23| 2| 2| China|\n+---+----+---+----+----+------+\n\n+---+-----+---+----+----+------+\n| id| name|age|c_id|c_id|c_name|\n+---+-----+---+----+----+------+\n| 1| andy| 20| 1| 1| USA|\n| 3|james| 18| 3|null| null|\n| 2| jeff| 23| 2| 2| China|\n+---+-----+---+----+----+------+\n\n" |
| } |
| ] |
| }, |
| "apps": [], |
| "runtimeInfos": { |
| "jobUrl": { |
| "propertyName": "jobUrl", |
| "label": "SPARK JOB", |
| "tooltip": "View in Spark web UI", |
| "group": "spark", |
| "values": [ |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d113" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d114" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d115" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d116" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d117" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d118" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d119" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d120" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d121" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d122" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d123" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d124" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d125" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d126" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d127" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d128" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d129" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d130" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d131" |
| } |
| ], |
| "interpreterSettingId": "spark" |
| } |
| }, |
| "progressUpdateIntervalMs": 500, |
| "jobName": "paragraph_1578387700851_-770209064", |
| "id": "20180530-130126_1642948432", |
| "dateCreated": "2020-01-07 17:01:40.851", |
| "dateStarted": "2021-07-26 04:33:59.566", |
| "dateFinished": "2021-07-26 04:34:03.252", |
| "status": "FINISHED" |
| }, |
| { |
| "title": "Join on Multiple Fields", |
| "text": "%spark.pyspark\n\ndf1 \u003d spark.createDataFrame([(\"andy\", 20, 1, 1), (\"jeff\", 23, 1, 2), (\"james\", 12, 2, 2)]).toDF(\"name\", \"age\", \"key_1\", \"key_2\")\ndf1.show()\n\ndf2 \u003d spark.createDataFrame([(1, 1, \"USA\"), (2, 2, \"China\")]).toDF(\"key_1\", \"key_2\", \"country\")\ndf2.show()\n\n# Join on 2 fields: key_1, key_2\n\n# You can pass a list of field name if the join field names are the same in both tables\ndf3 \u003d df1.join(df2, [\"key_1\", \"key_2\"])\ndf3.show()\n\n# Or you can specify the join condition expclitly in case when the join fields name is differetnt in the two tables\ndf4 \u003d df1.join(df2, (df1[\"key_1\"] \u003d\u003d df2[\"key_1\"]) \u0026 (df1[\"key_2\"] \u003d\u003d df2[\"key_2\"]))\ndf4.show()\n\n", |
| "user": "anonymous", |
| "dateUpdated": "2021-07-26 04:34:03.269", |
| "progress": 27, |
| "config": { |
| "editorSetting": { |
| "language": "python", |
| "editOnDblClick": false, |
| "completionKey": "TAB", |
| "completionSupport": true |
| }, |
| "colWidth": 6.0, |
| "editorMode": "ace/mode/python", |
| "fontSize": 9.0, |
| "title": true, |
| "results": {}, |
| "enabled": true, |
| "runOnSelectionChange": true, |
| "checkEmpty": true |
| }, |
| "settings": { |
| "params": {}, |
| "forms": {} |
| }, |
| "results": { |
| "code": "SUCCESS", |
| "msg": [ |
| { |
| "type": "TEXT", |
| "data": "+-----+---+-----+-----+\n| name|age|key_1|key_2|\n+-----+---+-----+-----+\n| andy| 20| 1| 1|\n| jeff| 23| 1| 2|\n|james| 12| 2| 2|\n+-----+---+-----+-----+\n\n+-----+-----+-------+\n|key_1|key_2|country|\n+-----+-----+-------+\n| 1| 1| USA|\n| 2| 2| China|\n+-----+-----+-------+\n\n+-----+-----+-----+---+-------+\n|key_1|key_2| name|age|country|\n+-----+-----+-----+---+-------+\n| 1| 1| andy| 20| USA|\n| 2| 2|james| 12| China|\n+-----+-----+-----+---+-------+\n\n+-----+---+-----+-----+-----+-----+-------+\n| name|age|key_1|key_2|key_1|key_2|country|\n+-----+---+-----+-----+-----+-----+-------+\n| andy| 20| 1| 1| 1| 1| USA|\n|james| 12| 2| 2| 2| 2| China|\n+-----+---+-----+-----+-----+-----+-------+\n\n" |
| } |
| ] |
| }, |
| "apps": [], |
| "runtimeInfos": { |
| "jobUrl": { |
| "propertyName": "jobUrl", |
| "label": "SPARK JOB", |
| "tooltip": "View in Spark web UI", |
| "group": "spark", |
| "values": [ |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d132" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d133" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d134" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d135" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d136" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d137" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d138" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d139" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d140" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d141" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d142" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d143" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d144" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d145" |
| } |
| ], |
| "interpreterSettingId": "spark" |
| } |
| }, |
| "progressUpdateIntervalMs": 500, |
| "jobName": "paragraph_1578387700851_-177297320", |
| "id": "20180530-135600_354945835", |
| "dateCreated": "2020-01-07 17:01:40.851", |
| "dateStarted": "2021-07-26 04:34:03.273", |
| "dateFinished": "2021-07-26 04:34:05.760", |
| "status": "FINISHED" |
| }, |
| { |
| "title": "Use SQL directly", |
| "text": "%spark.pyspark\n\ndf1 \u003d spark.createDataFrame([(1, \"andy\", 20, \"USA\"), (2, \"jeff\", 23, \"China\"), (3, \"james\", 18, \"USA\")]) \\\n .toDF(\"id\", \"name\", \"age\", \"country\")\n# call createOrReplaceTempView first if you want to query this DataFrame via sql\ndf1.createOrReplaceTempView(\"people\")\n# SparkSession.sql return DataFrame\ndf2 \u003d spark.sql(\"select name, age from people\")\ndf2.show()\n\n# You need to register udf if you want to use it in sql\nspark.udf.register(\"udf1\", lambda e : e.upper())\ndf3 \u003d spark.sql(\"select udf1(name), age from people\")\ndf3.show()", |
| "user": "anonymous", |
| "dateUpdated": "2021-07-26 04:34:05.774", |
| "progress": 0, |
| "config": { |
| "editorSetting": { |
| "language": "python", |
| "editOnDblClick": false, |
| "completionKey": "TAB", |
| "completionSupport": true |
| }, |
| "colWidth": 12.0, |
| "editorMode": "ace/mode/python", |
| "fontSize": 9.0, |
| "title": true, |
| "results": {}, |
| "enabled": true, |
| "runOnSelectionChange": true, |
| "checkEmpty": true |
| }, |
| "settings": { |
| "params": {}, |
| "forms": {} |
| }, |
| "results": { |
| "code": "SUCCESS", |
| "msg": [ |
| { |
| "type": "TEXT", |
| "data": "+-----+---+\n| name|age|\n+-----+---+\n| andy| 20|\n| jeff| 23|\n|james| 18|\n+-----+---+\n\n+----------+---+\n|udf1(name)|age|\n+----------+---+\n| ANDY| 20|\n| JEFF| 23|\n| JAMES| 18|\n+----------+---+\n\n" |
| } |
| ] |
| }, |
| "apps": [], |
| "runtimeInfos": { |
| "jobUrl": { |
| "propertyName": "jobUrl", |
| "label": "SPARK JOB", |
| "tooltip": "View in Spark web UI", |
| "group": "spark", |
| "values": [ |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d146" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d147" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d148" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d149" |
| } |
| ], |
| "interpreterSettingId": "spark" |
| } |
| }, |
| "progressUpdateIntervalMs": 500, |
| "jobName": "paragraph_1578387700851_1756979054", |
| "id": "20180530-132023_995737505", |
| "dateCreated": "2020-01-07 17:01:40.851", |
| "dateStarted": "2021-07-26 04:34:05.777", |
| "dateFinished": "2021-07-26 04:34:06.449", |
| "status": "FINISHED" |
| }, |
| { |
| "title": "Show Tables", |
| "text": "%spark.sql\n\nshow tables", |
| "user": "anonymous", |
| "dateUpdated": "2021-07-26 04:34:06.478", |
| "progress": 0, |
| "config": { |
| "runOnSelectionChange": true, |
| "title": true, |
| "checkEmpty": true, |
| "colWidth": 12.0, |
| "fontSize": 9.0, |
| "enabled": true, |
| "results": { |
| "0": { |
| "graph": { |
| "mode": "table", |
| "height": 300.0, |
| "optionOpen": false, |
| "setting": { |
| "table": { |
| "tableGridState": {}, |
| "tableColumnTypeState": { |
| "names": { |
| "database": "string", |
| "tableName": "string", |
| "isTemporary": "string" |
| }, |
| "updated": false |
| }, |
| "tableOptionSpecHash": "[{\"name\":\"useFilter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable filter for columns\"},{\"name\":\"showPagination\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable pagination for better navigation\"},{\"name\":\"showAggregationFooter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable a footer for displaying aggregated values\"}]", |
| "tableOptionValue": { |
| "useFilter": false, |
| "showPagination": false, |
| "showAggregationFooter": false |
| }, |
| "updated": false, |
| "initialized": false |
| } |
| }, |
| "commonSetting": {} |
| } |
| } |
| }, |
| "editorSetting": { |
| "language": "sql", |
| "editOnDblClick": false, |
| "completionKey": "TAB", |
| "completionSupport": true |
| }, |
| "editorMode": "ace/mode/sql" |
| }, |
| "settings": { |
| "params": {}, |
| "forms": {} |
| }, |
| "results": { |
| "code": "SUCCESS", |
| "msg": [ |
| { |
| "type": "TABLE", |
| "data": "database\ttableName\tisTemporary\n\tbank\ttrue\n\tpeople\ttrue\n" |
| } |
| ] |
| }, |
| "apps": [], |
| "runtimeInfos": {}, |
| "progressUpdateIntervalMs": 500, |
| "jobName": "paragraph_1578388432752_877455036", |
| "id": "paragraph_1578388432752_877455036", |
| "dateCreated": "2020-01-07 17:13:52.752", |
| "dateStarted": "2021-07-26 04:34:06.482", |
| "dateFinished": "2021-07-26 04:34:06.556", |
| "status": "FINISHED" |
| }, |
| { |
| "title": "Visualize DataFrame/Dataset", |
| "text": "%md\n\nThere\u0027s 2 approaches to visualize DataFrame/Dataset in Zeppelin\n\n* Use SparkSQLInterpreter via `%spark.sql`\n* Use ZeppelinContext via `z.show`\n\n", |
| "user": "anonymous", |
| "dateUpdated": "2021-07-26 04:34:06.582", |
| "progress": 0, |
| "config": { |
| "tableHide": false, |
| "editorSetting": { |
| "language": "markdown", |
| "editOnDblClick": true, |
| "completionKey": "TAB", |
| "completionSupport": false |
| }, |
| "colWidth": 12.0, |
| "editorMode": "ace/mode/markdown", |
| "fontSize": 9.0, |
| "editorHide": true, |
| "title": false, |
| "results": {}, |
| "enabled": true, |
| "runOnSelectionChange": true, |
| "checkEmpty": true |
| }, |
| "settings": { |
| "params": {}, |
| "forms": {} |
| }, |
| "results": { |
| "code": "SUCCESS", |
| "msg": [ |
| { |
| "type": "HTML", |
| "data": "\u003cdiv class\u003d\"markdown-body\"\u003e\n\u003cp\u003eThere\u0026rsquo;s 2 approaches to visualize DataFrame/Dataset in Zeppelin\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eUse SparkSQLInterpreter via \u003ccode\u003e%spark.sql\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003eUse ZeppelinContext via \u003ccode\u003ez.show\u003c/code\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\n\u003c/div\u003e" |
| } |
| ] |
| }, |
| "apps": [], |
| "runtimeInfos": {}, |
| "progressUpdateIntervalMs": 500, |
| "jobName": "paragraph_1578387700851_1924561483", |
| "id": "20180530-132128_2114955642", |
| "dateCreated": "2020-01-07 17:01:40.851", |
| "dateStarted": "2021-07-26 04:34:06.587", |
| "dateFinished": "2021-07-26 04:34:06.602", |
| "status": "FINISHED" |
| }, |
| { |
| "title": "Visualize DataFrame/Dataset via z.show", |
| "text": "%spark.pyspark\n\ndf1 \u003d spark.createDataFrame([(1, \"andy\", 20, \"USA\"), (2, \"jeff\", 23, \"China\"), (3, \"james\", 18, \"USA\")]).toDF(\"id\", \"name\", \"age\", \"country\")\ndf2 \u003d df1.groupBy(\"country\").count()\nz.show(df2)\n\n\n\n\n\n\n", |
| "user": "anonymous", |
| "dateUpdated": "2021-07-26 04:34:06.687", |
| "progress": 0, |
| "config": { |
| "editorSetting": { |
| "language": "python", |
| "editOnDblClick": false, |
| "completionKey": "TAB", |
| "completionSupport": true |
| }, |
| "colWidth": 6.0, |
| "editorMode": "ace/mode/python", |
| "fontSize": 9.0, |
| "title": true, |
| "results": { |
| "0": { |
| "graph": { |
| "mode": "multiBarChart", |
| "height": 300.0, |
| "optionOpen": false, |
| "setting": { |
| "table": { |
| "tableGridState": {}, |
| "tableColumnTypeState": { |
| "names": { |
| "country": "string", |
| "count": "string" |
| }, |
| "updated": false |
| }, |
| "tableOptionSpecHash": "[{\"name\":\"useFilter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable filter for columns\"},{\"name\":\"showPagination\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable pagination for better navigation\"},{\"name\":\"showAggregationFooter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable a footer for displaying aggregated values\"}]", |
| "tableOptionValue": { |
| "useFilter": false, |
| "showPagination": false, |
| "showAggregationFooter": false |
| }, |
| "updated": false, |
| "initialized": false |
| }, |
| "multiBarChart": { |
| "rotate": { |
| "degree": "-45" |
| }, |
| "xLabelStatus": "default" |
| } |
| }, |
| "commonSetting": {}, |
| "keys": [ |
| { |
| "name": "country", |
| "index": 0.0, |
| "aggr": "sum" |
| } |
| ], |
| "groups": [], |
| "values": [ |
| { |
| "name": "count", |
| "index": 1.0, |
| "aggr": "sum" |
| } |
| ] |
| }, |
| "helium": {} |
| }, |
| "1": { |
| "graph": { |
| "mode": "multiBarChart", |
| "height": 300.0, |
| "optionOpen": false, |
| "setting": { |
| "table": { |
| "tableGridState": {}, |
| "tableColumnTypeState": { |
| "names": { |
| "country": "string", |
| "count": "string" |
| }, |
| "updated": false |
| }, |
| "tableOptionSpecHash": "[{\"name\":\"useFilter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable filter for columns\"},{\"name\":\"showPagination\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable pagination for better navigation\"},{\"name\":\"showAggregationFooter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable a footer for displaying aggregated values\"}]", |
| "tableOptionValue": { |
| "useFilter": false, |
| "showPagination": false, |
| "showAggregationFooter": false |
| }, |
| "updated": false, |
| "initialized": false |
| }, |
| "multiBarChart": { |
| "rotate": { |
| "degree": "-45" |
| }, |
| "xLabelStatus": "default" |
| } |
| }, |
| "commonSetting": {}, |
| "keys": [ |
| { |
| "name": "country", |
| "index": 0.0, |
| "aggr": "sum" |
| } |
| ], |
| "groups": [], |
| "values": [ |
| { |
| "name": "count", |
| "index": 1.0, |
| "aggr": "sum" |
| } |
| ] |
| }, |
| "helium": {} |
| } |
| }, |
| "enabled": true, |
| "runOnSelectionChange": true, |
| "checkEmpty": true |
| }, |
| "settings": { |
| "params": {}, |
| "forms": {} |
| }, |
| "results": { |
| "code": "SUCCESS", |
| "msg": [ |
| { |
| "type": "TABLE", |
| "data": "country\tcount\nChina\t1\nUSA\t2\n" |
| } |
| ] |
| }, |
| "apps": [], |
| "runtimeInfos": { |
| "jobUrl": { |
| "propertyName": "jobUrl", |
| "label": "SPARK JOB", |
| "tooltip": "View in Spark web UI", |
| "group": "spark", |
| "values": [ |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d150" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d151" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d152" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d153" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d154" |
| } |
| ], |
| "interpreterSettingId": "spark" |
| } |
| }, |
| "progressUpdateIntervalMs": 500, |
| "jobName": "paragraph_1578387700851_-809695439", |
| "id": "20180530-132634_1285621466", |
| "dateCreated": "2020-01-07 17:01:40.851", |
| "dateStarted": "2021-07-26 04:34:06.702", |
| "dateFinished": "2021-07-26 04:34:07.612", |
| "status": "FINISHED" |
| }, |
| { |
| "title": "Visualize DataFrame/Dataset via %spark.sql", |
| "text": "%spark.pyspark\n\ndf1 \u003d spark.createDataFrame([(1, \"andy\", 20, \"USA\"), (2, \"jeff\", 23, \"China\"), (3, \"james\", 18, \"USA\")]) \\\n .toDF(\"id\", \"name\", \"age\", \"country\")\n \n# register this DataFrame first before querying it via %spark.sql\ndf1.createOrReplaceTempView(\"people\")", |
| "user": "anonymous", |
| "dateUpdated": "2021-07-26 04:34:07.700", |
| "progress": 0, |
| "config": { |
| "editorSetting": { |
| "language": "python", |
| "editOnDblClick": false, |
| "completionKey": "TAB", |
| "completionSupport": true |
| }, |
| "colWidth": 6.0, |
| "editorMode": "ace/mode/python", |
| "fontSize": 9.0, |
| "title": true, |
| "results": {}, |
| "enabled": true, |
| "runOnSelectionChange": true, |
| "checkEmpty": true |
| }, |
| "settings": { |
| "params": {}, |
| "forms": {} |
| }, |
| "results": { |
| "code": "SUCCESS", |
| "msg": [] |
| }, |
| "apps": [], |
| "runtimeInfos": {}, |
| "progressUpdateIntervalMs": 500, |
| "jobName": "paragraph_1578387700851_381269276", |
| "id": "20180530-132657_668624333", |
| "dateCreated": "2020-01-07 17:01:40.851", |
| "dateStarted": "2021-07-26 04:34:07.704", |
| "dateFinished": "2021-07-26 04:34:07.979", |
| "status": "FINISHED" |
| }, |
| { |
| "title": "", |
| "text": "%spark.sql\n\nselect country, count(1) as count from people group by country", |
| "user": "anonymous", |
| "dateUpdated": "2021-07-26 04:34:08.003", |
| "progress": 0, |
| "config": { |
| "editorSetting": { |
| "language": "sql", |
| "editOnDblClick": false, |
| "completionKey": "TAB", |
| "completionSupport": true |
| }, |
| "colWidth": 6.0, |
| "editorMode": "ace/mode/sql", |
| "fontSize": 9.0, |
| "results": { |
| "0": { |
| "graph": { |
| "mode": "multiBarChart", |
| "height": 300.0, |
| "optionOpen": false, |
| "setting": { |
| "table": { |
| "tableGridState": { |
| "columns": [ |
| { |
| "name": "country", |
| "visible": true, |
| "width": "*", |
| "sort": {}, |
| "filters": [ |
| {} |
| ], |
| "pinned": "" |
| }, |
| { |
| "name": "count", |
| "visible": true, |
| "width": "*", |
| "sort": {}, |
| "filters": [ |
| {} |
| ], |
| "pinned": "" |
| } |
| ], |
| "scrollFocus": {}, |
| "selection": [], |
| "grouping": { |
| "grouping": [], |
| "aggregations": [], |
| "rowExpandedStates": {} |
| }, |
| "treeView": {}, |
| "pagination": { |
| "paginationCurrentPage": 1.0, |
| "paginationPageSize": 250.0 |
| } |
| }, |
| "tableColumnTypeState": { |
| "names": { |
| "country": "string", |
| "count": "number" |
| }, |
| "updated": false |
| }, |
| "tableOptionSpecHash": "[{\"name\":\"useFilter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable filter for columns\"},{\"name\":\"showPagination\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable pagination for better navigation\"},{\"name\":\"showAggregationFooter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable a footer for displaying aggregated values\"}]", |
| "tableOptionValue": { |
| "useFilter": false, |
| "showPagination": false, |
| "showAggregationFooter": false |
| }, |
| "updated": false, |
| "initialized": false |
| }, |
| "multiBarChart": { |
| "rotate": { |
| "degree": "-45" |
| }, |
| "xLabelStatus": "default", |
| "stacked": false |
| }, |
| "stackedAreaChart": { |
| "rotate": { |
| "degree": "-45" |
| }, |
| "xLabelStatus": "default" |
| }, |
| "lineChart": { |
| "rotate": { |
| "degree": "-45" |
| }, |
| "xLabelStatus": "default" |
| } |
| }, |
| "commonSetting": {}, |
| "keys": [ |
| { |
| "name": "country", |
| "index": 0.0, |
| "aggr": "sum" |
| } |
| ], |
| "groups": [], |
| "values": [ |
| { |
| "name": "count", |
| "index": 1.0, |
| "aggr": "sum" |
| } |
| ] |
| }, |
| "helium": {} |
| } |
| }, |
| "enabled": true, |
| "runOnSelectionChange": true, |
| "title": false, |
| "checkEmpty": true |
| }, |
| "settings": { |
| "params": {}, |
| "forms": {} |
| }, |
| "results": { |
| "code": "SUCCESS", |
| "msg": [ |
| { |
| "type": "TABLE", |
| "data": "country\tcount\nChina\t1\nUSA\t2\n" |
| } |
| ] |
| }, |
| "apps": [], |
| "runtimeInfos": { |
| "jobUrl": { |
| "propertyName": "jobUrl", |
| "label": "SPARK JOB", |
| "tooltip": "View in Spark web UI", |
| "group": "spark", |
| "values": [ |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d155" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d156" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d157" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d158" |
| }, |
| { |
| "jobUrl": "http://3745c7ed824d:4040/jobs/job?id\u003d159" |
| } |
| ], |
| "interpreterSettingId": "spark" |
| } |
| }, |
| "progressUpdateIntervalMs": 500, |
| "jobName": "paragraph_1578387700851_601200360", |
| "id": "20180530-132823_944494152", |
| "dateCreated": "2020-01-07 17:01:40.851", |
| "dateStarted": "2021-07-26 04:34:08.006", |
| "dateFinished": "2021-07-26 04:34:08.652", |
| "status": "FINISHED" |
| }, |
| { |
| "title": "", |
| "text": "%spark.sql\n", |
| "user": "anonymous", |
| "dateUpdated": "2021-07-26 04:34:08.707", |
| "progress": 0, |
| "config": { |
| "editorSetting": { |
| "language": "sql", |
| "editOnDblClick": false, |
| "completionKey": "TAB", |
| "completionSupport": true |
| }, |
| "colWidth": 12.0, |
| "editorMode": "ace/mode/sql", |
| "fontSize": 9.0, |
| "results": {}, |
| "enabled": true |
| }, |
| "settings": { |
| "params": {}, |
| "forms": {} |
| }, |
| "results": { |
| "code": "SUCCESS", |
| "msg": [] |
| }, |
| "apps": [], |
| "runtimeInfos": {}, |
| "progressUpdateIntervalMs": 500, |
| "jobName": "paragraph_1578387700851_-2136199261", |
| "id": "20180530-132849_1305166760", |
| "dateCreated": "2020-01-07 17:01:40.851", |
| "dateStarted": "2021-07-26 04:34:08.710", |
| "dateFinished": "2021-07-26 04:34:08.716", |
| "status": "FINISHED" |
| } |
| ], |
| "name": "3. Spark SQL (PySpark)", |
| "id": "2EWM84JXA", |
| "defaultInterpreterGroup": "spark", |
| "version": "0.9.0-SNAPSHOT", |
| "noteParams": {}, |
| "noteForms": {}, |
| "angularObjects": {}, |
| "config": { |
| "isZeppelinNotebookCronEnable": false |
| }, |
| "info": { |
| "isRunning": false |
| } |
| } |
