| { |
| "paragraphs": [ |
| { |
| "title": "Introduction", |
| "text": "%md\n\nThis is a tutorial note for Flink batch scenario (`To be noticed, you need to use flink 1.9 or afterwards`), . You can run flink scala api via `%flink` and run flink batch sql via `%flink.bsql`. \nThis note use flink\u0027s DataSet api to demonstrate flink\u0027s batch capablity. DataSet is only supported by flink planner, so here we have to specify the planner `zeppelin.flink.planner` as `flink`, otherwise it would use blink planner by default which doesn\u0027t support DataSet api.\n", |
| "user": "anonymous", |
| "dateUpdated": "2019-10-08 15:15:54.910", |
| "config": { |
| "tableHide": false, |
| "editorSetting": { |
| "language": "markdown", |
| "editOnDblClick": true, |
| "completionKey": "TAB", |
| "completionSupport": false |
| }, |
| "colWidth": 12.0, |
| "editorMode": "ace/mode/markdown", |
| "fontSize": 9.0, |
| "editorHide": true, |
| "title": false, |
| "runOnSelectionChange": true, |
| "checkEmpty": true, |
| "results": {}, |
| "enabled": true |
| }, |
| "settings": { |
| "params": {}, |
| "forms": {} |
| }, |
| "results": { |
| "code": "SUCCESS", |
| "msg": [ |
| { |
| "type": "HTML", |
| "data": "\u003cdiv class\u003d\"markdown-body\"\u003e\n\u003cp\u003eThis is a tutorial note for Flink batch scenario (\u003ccode\u003eTo be noticed, you need to use flink 1.9 or afterwards\u003c/code\u003e), . You can run flink scala api via \u003ccode\u003e%flink\u003c/code\u003e and run flink batch sql via \u003ccode\u003e%flink.bsql\u003c/code\u003e.\u003cbr/\u003eThis note use flink\u0026rsquo;s DataSet api to demonstrate flink\u0026rsquo;s batch capablity. DataSet is only supported by flink planner, so here we have to specify the planner \u003ccode\u003ezeppelin.flink.planner\u003c/code\u003e as \u003ccode\u003eflink\u003c/code\u003e, otherwise it would use blink planner by default which doesn\u0026rsquo;t support DataSet api.\u003c/p\u003e\n\u003c/div\u003e" |
| } |
| ] |
| }, |
| "apps": [], |
| "progressUpdateIntervalMs": 500, |
| "jobName": "paragraph_1569489641095_-188362229", |
| "id": "paragraph_1547794482637_957545547", |
| "dateCreated": "2019-09-26 17:20:41.095", |
| "dateStarted": "2019-10-08 15:15:54.910", |
| "dateFinished": "2019-10-08 15:15:54.923", |
| "status": "FINISHED" |
| }, |
| { |
| "title": "Configure Flink Interpreter", |
| "text": "%flink.conf\n\nFLINK_HOME \u003cFLINK_INSTALLATION\u003e\n# DataSet is only supported in flink planner, so here we use flink planner. By default it is blink planner\nzeppelin.flink.planner flink\n\n\n", |
| "user": "anonymous", |
| "dateUpdated": "2019-10-11 10:37:39.948", |
| "config": { |
| "editorSetting": { |
| "language": "text", |
| "editOnDblClick": false, |
| "completionKey": "TAB", |
| "completionSupport": true |
| }, |
| "colWidth": 12.0, |
| "editorMode": "ace/mode/text", |
| "fontSize": 9.0, |
| "editorHide": false, |
| "title": false, |
| "runOnSelectionChange": true, |
| "checkEmpty": true, |
| "results": {}, |
| "enabled": true |
| }, |
| "settings": { |
| "params": {}, |
| "forms": {} |
| }, |
| "results": { |
| "code": "SUCCESS", |
| "msg": [] |
| }, |
| "apps": [], |
| "progressUpdateIntervalMs": 500, |
| "jobName": "paragraph_1569489641097_1856810430", |
| "id": "paragraph_1546565092490_1952685806", |
| "dateCreated": "2019-09-26 17:20:41.097", |
| "dateStarted": "2019-10-11 10:00:42.031", |
| "dateFinished": "2019-10-11 10:00:42.052", |
| "status": "FINISHED" |
| }, |
| { |
| "text": "%sh\n\ncd /tmp\nwget https://s3.amazonaws.com/apache-zeppelin/tutorial/bank/bank.csv\n", |
| "user": "anonymous", |
| "dateUpdated": "2019-10-08 15:16:45.127", |
| "config": { |
| "runOnSelectionChange": true, |
| "title": false, |
| "checkEmpty": true, |
| "colWidth": 12.0, |
| "fontSize": 9.0, |
| "enabled": true, |
| "results": {}, |
| "editorSetting": { |
| "language": "sh", |
| "editOnDblClick": false, |
| "completionKey": "TAB", |
| "completionSupport": false |
| }, |
| "editorMode": "ace/mode/sh" |
| }, |
| "settings": { |
| "params": {}, |
| "forms": {} |
| }, |
| "results": { |
| "code": "SUCCESS", |
| "msg": [ |
| { |
| "type": "TEXT", |
| "data": "--2019-10-08 15:16:46-- https://s3.amazonaws.com/apache-zeppelin/tutorial/bank/bank.csv\nResolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.120.2\nConnecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.120.2|:443... connected.\nHTTP request sent, awaiting response... 200 OK\nLength: 461474 (451K) [application/octet-stream]\nSaving to: \u0027bank.csv\u0027\n\n 0K .......... .......... .......... .......... .......... 11% 91.0K 4s\n 50K .......... .......... .......... .......... .......... 22% 568K 2s\n 100K .......... .......... .......... .......... .......... 33% 372K 2s\n 150K .......... .......... .......... .......... .......... 44% 176K 1s\n 200K .......... .......... .......... .......... .......... 55% 626K 1s\n 250K .......... .......... .......... .......... .......... 66% 326K 1s\n 300K .......... .......... .......... .......... .......... 77% 433K 0s\n 350K .......... .......... .......... .......... .......... 88% 77.0K 0s\n 400K .......... .......... .......... .......... .......... 99% 77.4M 0s\n 450K 100% 13.2K\u003d2.1s\n\n2019-10-08 15:16:50 (219 KB/s) - \u0027bank.csv\u0027 saved [461474/461474]\n\n" |
| } |
| ] |
| }, |
| "apps": [], |
| "progressUpdateIntervalMs": 500, |
| "jobName": "paragraph_1569489915993_-1377803690", |
| "id": "paragraph_1569489915993_-1377803690", |
| "dateCreated": "2019-09-26 17:25:15.993", |
| "dateStarted": "2019-10-08 15:16:45.132", |
| "dateFinished": "2019-10-08 15:16:50.387", |
| "status": "FINISHED" |
| }, |
| { |
| "title": "Load Bank Data", |
| "text": "%flink\n\nval bankText \u003d benv.readTextFile(\"/tmp/bank.csv\")\nval bank \u003d bankText.map(s \u003d\u003e s.split(\";\")).filter(s \u003d\u003e s(0) !\u003d \"\\\"age\\\"\").map(\n s \u003d\u003e (s(0).toInt,\n s(1).replaceAll(\"\\\"\", \"\"),\n s(2).replaceAll(\"\\\"\", \"\"),\n s(3).replaceAll(\"\\\"\", \"\"),\n s(5).replaceAll(\"\\\"\", \"\").toInt\n )\n )\n\nbtenv.registerDataSet(\"bank\", bank, \u0027age, \u0027job, \u0027marital, \u0027education, \u0027balance)\n\n\n", |
| "user": "anonymous", |
| "dateUpdated": "2019-10-11 10:01:44.897", |
| "config": { |
| "editorSetting": { |
| "language": "scala", |
| "editOnDblClick": false, |
| "completionKey": "TAB", |
| "completionSupport": true |
| }, |
| "colWidth": 12.0, |
| "editorMode": "ace/mode/scala", |
| "fontSize": 9.0, |
| "title": false, |
| "runOnSelectionChange": true, |
| "checkEmpty": true, |
| "results": { |
| "0": { |
| "graph": { |
| "mode": "table", |
| "height": 111.0, |
| "optionOpen": false |
| } |
| } |
| }, |
| "enabled": true |
| }, |
| "settings": { |
| "params": {}, |
| "forms": {} |
| }, |
| "results": { |
| "code": "SUCCESS", |
| "msg": [ |
| { |
| "type": "TEXT", |
| "data": "\u001b[1m\u001b[34mbankText\u001b[0m: \u001b[1m\u001b[32morg.apache.flink.api.scala.DataSet[String]\u001b[0m \u003d org.apache.flink.api.scala.DataSet@683ccfd\n\u001b[1m\u001b[34mbank\u001b[0m: \u001b[1m\u001b[32morg.apache.flink.api.scala.DataSet[(Int, String, String, String, Int)]\u001b[0m \u003d org.apache.flink.api.scala.DataSet@35e6c08f\n" |
| } |
| ] |
| }, |
| "apps": [], |
| "progressUpdateIntervalMs": 500, |
| "jobName": "paragraph_1569489641099_-2030350664", |
| "id": "paragraph_1546584347815_1533642635", |
| "dateCreated": "2019-09-26 17:20:41.099", |
| "dateStarted": "2019-10-11 10:01:44.904", |
| "dateFinished": "2019-10-11 10:02:12.333", |
| "status": "FINISHED" |
| }, |
| { |
| "text": "%flink.bsql\n\ndescribe bank\n\n\n\n\n\n\n\n\n", |
| "user": "anonymous", |
| "dateUpdated": "2019-10-11 10:07:51.800", |
| "config": { |
| "editorSetting": { |
| "language": "sql", |
| "editOnDblClick": false, |
| "completionKey": "TAB", |
| "completionSupport": true |
| }, |
| "colWidth": 6.0, |
| "editorMode": "ace/mode/sql", |
| "fontSize": 9.0, |
| "runOnSelectionChange": true, |
| "title": false, |
| "checkEmpty": true, |
| "results": { |
| "0": { |
| "graph": { |
| "mode": "table", |
| "height": 300.0, |
| "optionOpen": false, |
| "setting": { |
| "table": { |
| "tableGridState": {}, |
| "tableColumnTypeState": { |
| "names": { |
| "table": "string" |
| }, |
| "updated": false |
| }, |
| "tableOptionSpecHash": "[{\"name\":\"useFilter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable filter for columns\"},{\"name\":\"showPagination\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable pagination for better navigation\"},{\"name\":\"showAggregationFooter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable a footer for displaying aggregated values\"}]", |
| "tableOptionValue": { |
| "useFilter": false, |
| "showPagination": false, |
| "showAggregationFooter": false |
| }, |
| "updated": false, |
| "initialized": false |
| } |
| }, |
| "commonSetting": {} |
| } |
| } |
| }, |
| "enabled": true |
| }, |
| "settings": { |
| "params": {}, |
| "forms": {} |
| }, |
| "results": { |
| "code": "SUCCESS", |
| "msg": [ |
| { |
| "type": "TEXT", |
| "data": "Column\tType\nOptional[age]\tOptional[INT]\nOptional[job]\tOptional[STRING]\nOptional[marital]\tOptional[STRING]\nOptional[education]\tOptional[STRING]\nOptional[balance]\tOptional[INT]\n" |
| } |
| ] |
| }, |
| "apps": [], |
| "progressUpdateIntervalMs": 500, |
| "jobName": "paragraph_1569491358206_1385745589", |
| "id": "paragraph_1569491358206_1385745589", |
| "dateCreated": "2019-09-26 17:49:18.206", |
| "dateStarted": "2019-10-11 10:07:51.808", |
| "dateFinished": "2019-10-11 10:07:52.076", |
| "status": "FINISHED" |
| }, |
| { |
| "text": "%flink.bsql\n\nselect age, count(1) as v\nfrom bank \nwhere age \u003c 30 \ngroup by age \norder by age", |
| "user": "anonymous", |
| "dateUpdated": "2019-10-11 10:07:53.645", |
| "config": { |
| "editorSetting": { |
| "language": "sql", |
| "editOnDblClick": false, |
| "completionKey": "TAB", |
| "completionSupport": true |
| }, |
| "colWidth": 6.0, |
| "editorMode": "ace/mode/sql", |
| "fontSize": 9.0, |
| "runOnSelectionChange": true, |
| "title": false, |
| "checkEmpty": true, |
| "results": { |
| "0": { |
| "graph": { |
| "mode": "table", |
| "height": 300.0, |
| "optionOpen": false, |
| "setting": { |
| "table": { |
| "tableGridState": {}, |
| "tableColumnTypeState": { |
| "names": { |
| "age": "string", |
| "v": "string" |
| }, |
| "updated": false |
| }, |
| "tableOptionSpecHash": "[{\"name\":\"useFilter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable filter for columns\"},{\"name\":\"showPagination\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable pagination for better navigation\"},{\"name\":\"showAggregationFooter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable a footer for displaying aggregated values\"}]", |
| "tableOptionValue": { |
| "useFilter": false, |
| "showPagination": false, |
| "showAggregationFooter": false |
| }, |
| "updated": false, |
| "initialized": false |
| } |
| }, |
| "commonSetting": {} |
| } |
| } |
| }, |
| "enabled": true |
| }, |
| "settings": { |
| "params": {}, |
| "forms": {} |
| }, |
| "results": { |
| "code": "SUCCESS", |
| "msg": [ |
| { |
| "type": "TABLE", |
| "data": "age\tv\n19\t4\n20\t3\n21\t7\n22\t9\n23\t20\n24\t24\n25\t44\n26\t77\n27\t94\n28\t103\n29\t97\n" |
| }, |
| { |
| "type": "TEXT", |
| "data": "" |
| } |
| ] |
| }, |
| "apps": [], |
| "progressUpdateIntervalMs": 500, |
| "jobName": "paragraph_1569491511585_-677666348", |
| "id": "paragraph_1569491511585_-677666348", |
| "dateCreated": "2019-09-26 17:51:51.585", |
| "dateStarted": "2019-10-11 10:07:53.654", |
| "dateFinished": "2019-10-11 10:08:10.439", |
| "status": "FINISHED" |
| }, |
| { |
| "text": "%flink.bsql\n\nselect age, count(1) as v \nfrom bank \nwhere age \u003c ${maxAge\u003d30} \ngroup by age \norder by age", |
| "user": "anonymous", |
| "dateUpdated": "2019-10-11 10:08:13.910", |
| "config": { |
| "editorSetting": { |
| "language": "sql", |
| "editOnDblClick": false, |
| "completionKey": "TAB", |
| "completionSupport": true |
| }, |
| "colWidth": 6.0, |
| "editorMode": "ace/mode/sql", |
| "fontSize": 9.0, |
| "runOnSelectionChange": true, |
| "title": false, |
| "checkEmpty": true, |
| "results": { |
| "0": { |
| "graph": { |
| "mode": "pieChart", |
| "height": 300.0, |
| "optionOpen": false, |
| "setting": { |
| "table": { |
| "tableGridState": {}, |
| "tableColumnTypeState": { |
| "names": { |
| "age": "string", |
| "v": "string" |
| }, |
| "updated": false |
| }, |
| "tableOptionSpecHash": "[{\"name\":\"useFilter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable filter for columns\"},{\"name\":\"showPagination\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable pagination for better navigation\"},{\"name\":\"showAggregationFooter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable a footer for displaying aggregated values\"}]", |
| "tableOptionValue": { |
| "useFilter": false, |
| "showPagination": false, |
| "showAggregationFooter": false |
| }, |
| "updated": false, |
| "initialized": false |
| }, |
| "multiBarChart": { |
| "rotate": { |
| "degree": "-45" |
| }, |
| "xLabelStatus": "default" |
| } |
| }, |
| "commonSetting": {}, |
| "keys": [ |
| { |
| "name": "age", |
| "index": 0.0, |
| "aggr": "sum" |
| } |
| ], |
| "groups": [], |
| "values": [ |
| { |
| "name": "v", |
| "index": 1.0, |
| "aggr": "sum" |
| } |
| ] |
| }, |
| "helium": {} |
| } |
| }, |
| "enabled": true |
| }, |
| "settings": { |
| "params": { |
| "maxAge": "40" |
| }, |
| "forms": { |
| "maxAge": { |
| "type": "TextBox", |
| "name": "maxAge", |
| "defaultValue": "30", |
| "hidden": false |
| } |
| } |
| }, |
| "results": { |
| "code": "SUCCESS", |
| "msg": [ |
| { |
| "type": "TABLE", |
| "data": "age\tv\n19\t4\n20\t3\n21\t7\n22\t9\n23\t20\n24\t24\n25\t44\n26\t77\n27\t94\n28\t103\n29\t97\n30\t150\n31\t199\n32\t224\n33\t186\n34\t231\n35\t180\n36\t188\n37\t161\n38\t159\n39\t130\n" |
| }, |
| { |
| "type": "TEXT", |
| "data": "" |
| } |
| ] |
| }, |
| "apps": [], |
| "progressUpdateIntervalMs": 500, |
| "jobName": "paragraph_1569489641100_115468969", |
| "id": "paragraph_1546592465802_-1181051373", |
| "dateCreated": "2019-09-26 17:20:41.100", |
| "dateStarted": "2019-10-11 10:08:13.923", |
| "dateFinished": "2019-10-11 10:08:14.876", |
| "status": "FINISHED" |
| }, |
| { |
| "text": "%flink.bsql\n\nselect age, count(1) as v \nfrom bank \nwhere marital\u003d\u0027${marital\u003dsingle,single|divorced|married}\u0027 \ngroup by age \norder by age", |
| "user": "anonymous", |
| "dateUpdated": "2019-10-11 10:37:59.493", |
| "config": { |
| "editorSetting": { |
| "language": "sql", |
| "editOnDblClick": false, |
| "completionKey": "TAB", |
| "completionSupport": true |
| }, |
| "colWidth": 6.0, |
| "editorMode": "ace/mode/sql", |
| "fontSize": 9.0, |
| "runOnSelectionChange": true, |
| "title": false, |
| "checkEmpty": true, |
| "results": { |
| "0": { |
| "graph": { |
| "mode": "multiBarChart", |
| "height": 300.0, |
| "optionOpen": false, |
| "setting": { |
| "table": { |
| "tableGridState": {}, |
| "tableColumnTypeState": { |
| "names": { |
| "age": "string", |
| "v": "string" |
| }, |
| "updated": false |
| }, |
| "tableOptionSpecHash": "[{\"name\":\"useFilter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable filter for columns\"},{\"name\":\"showPagination\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable pagination for better navigation\"},{\"name\":\"showAggregationFooter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable a footer for displaying aggregated values\"}]", |
| "tableOptionValue": { |
| "useFilter": false, |
| "showPagination": false, |
| "showAggregationFooter": false |
| }, |
| "updated": false, |
| "initialized": false |
| }, |
| "multiBarChart": { |
| "rotate": { |
| "degree": "-45" |
| }, |
| "xLabelStatus": "default" |
| }, |
| "lineChart": { |
| "rotate": { |
| "degree": "-45" |
| }, |
| "xLabelStatus": "default" |
| }, |
| "stackedAreaChart": { |
| "rotate": { |
| "degree": "-45" |
| }, |
| "xLabelStatus": "default" |
| } |
| }, |
| "commonSetting": {}, |
| "keys": [ |
| { |
| "name": "age", |
| "index": 0.0, |
| "aggr": "sum" |
| } |
| ], |
| "groups": [], |
| "values": [ |
| { |
| "name": "v", |
| "index": 1.0, |
| "aggr": "sum" |
| } |
| ] |
| }, |
| "helium": {} |
| } |
| }, |
| "enabled": true |
| }, |
| "settings": { |
| "params": { |
| "marital": "married" |
| }, |
| "forms": { |
| "marital": { |
| "type": "Select", |
| "options": [ |
| { |
| "value": "single" |
| }, |
| { |
| "value": "divorced" |
| }, |
| { |
| "value": "married" |
| } |
| ], |
| "name": "marital", |
| "defaultValue": "single", |
| "hidden": false |
| } |
| } |
| }, |
| "results": { |
| "code": "SUCCESS", |
| "msg": [ |
| { |
| "type": "TABLE", |
| "data": "age\tv\n23\t3\n24\t11\n25\t11\n26\t18\n27\t26\n28\t23\n29\t37\n30\t56\n31\t104\n32\t105\n33\t103\n34\t142\n35\t109\n36\t117\n37\t100\n38\t99\n39\t88\n40\t105\n41\t97\n42\t91\n43\t79\n44\t68\n45\t76\n46\t82\n47\t78\n48\t91\n49\t87\n50\t74\n51\t63\n52\t66\n53\t75\n54\t56\n55\t68\n56\t50\n57\t78\n58\t67\n59\t56\n60\t36\n61\t15\n62\t5\n63\t7\n64\t6\n65\t4\n66\t7\n67\t5\n68\t1\n69\t5\n70\t5\n71\t5\n72\t4\n73\t6\n74\t2\n75\t3\n76\t1\n77\t5\n78\t2\n79\t3\n80\t6\n81\t1\n83\t2\n86\t1\n87\t1\n" |
| }, |
| { |
| "type": "TEXT", |
| "data": "" |
| } |
| ] |
| }, |
| "apps": [], |
| "progressUpdateIntervalMs": 500, |
| "jobName": "paragraph_1569489641100_-1622522691", |
| "id": "paragraph_1546592478596_-1766740165", |
| "dateCreated": "2019-09-26 17:20:41.100", |
| "dateStarted": "2019-10-11 10:08:16.423", |
| "dateFinished": "2019-10-11 10:08:17.295", |
| "status": "FINISHED" |
| }, |
| { |
| "text": "\n", |
| "user": "anonymous", |
| "dateUpdated": "2019-09-26 17:54:48.292", |
| "config": { |
| "colWidth": 12.0, |
| "fontSize": 9.0, |
| "enabled": true, |
| "results": {}, |
| "editorSetting": { |
| "language": "scala", |
| "editOnDblClick": false, |
| "completionKey": "TAB", |
| "completionSupport": true |
| }, |
| "editorMode": "ace/mode/scala" |
| }, |
| "settings": { |
| "params": {}, |
| "forms": {} |
| }, |
| "apps": [], |
| "progressUpdateIntervalMs": 500, |
| "jobName": "paragraph_1569489641101_-1182720873", |
| "id": "paragraph_1553093710610_-1734599499", |
| "dateCreated": "2019-09-26 17:20:41.101", |
| "status": "READY" |
| } |
| ], |
| "name": "Flink Batch Tutorial", |
| "id": "2EN1E1ATY", |
| "defaultInterpreterGroup": "spark", |
| "version": "0.9.0-SNAPSHOT", |
| "permissions": {}, |
| "noteParams": {}, |
| "noteForms": {}, |
| "angularObjects": {}, |
| "config": { |
| "isZeppelinNotebookCronEnable": false |
| }, |
| "info": {}, |
| "path": "/Flink Batch Tutorial" |
| } |
