{% include JB/setup %}
Zeppelin supports python language which is very popular in data analytics and machine learning.
For beginner, we would suggest you to play Python in Zeppelin docker first. In the Zeppelin docker image, we have already installed miniconda and lots of useful python libraries including IPython's prerequisites, so %python
would use IPython.
Without any extra configuration, you can run most of tutorial notes under folder Python Tutorial
directly.
docker run -u $(id -u) -p 8080:8080 --rm --name zeppelin apache/zeppelin:0.10.0
After running the above command, you can open http://localhost:8080
to play Python in Zeppelin.
%python
)The vanilla python interpreter provides basic python interpreter feature, only python installed is required.
The vanilla python interpreter can display matplotlib figures inline automatically using the matplotlib
:
%python import matplotlib.pyplot as plt plt.plot([1, 2, 3])
The output of this command will by default be converted to HTML by implicitly making use of the %html
magic. Additional configuration can be achieved using the builtin z.configure_mpl()
method. For example,
z.configure_mpl(width=400, height=300, fmt='svg') plt.plot([1, 2, 3])
Will produce a 400x300 image in SVG format, which by default are normally 600x400 and PNG respectively. In the future, another option called angular
can be used to make it possible to update a plot produced from one paragraph directly from another (the output will be %angular
instead of %html
). However, this feature is already available in the pyspark
interpreter. More details can be found in the included “Zeppelin Tutorial: Python - matplotlib basic” tutorial notebook.
If Zeppelin cannot find the matplotlib backend files (which should usually be found in $ZEPPELIN_HOME/interpreter/lib/python
) in your PYTHONPATH
, then the backend will automatically be set to agg, and the (otherwise deprecated) instructions below can be used for more limited inline plotting.
If you are unable to load the inline backend, use z.show(plt)
:
%python import matplotlib.pyplot as plt plt.figure() (.. ..) z.show(plt) plt.close()
The z.show()
function can take optional parameters to adapt graph dimensions (width and height) as well as output format (png or optionally svg).
%python z.show(plt, width='50px') z.show(plt, height='150px', fmt='svg')
%python.ipython
) (recommended)IPython is more powerful than the vanilla python interpreter with extra functionality. This is what we recommend you to use instead of vanilla python interpreter. You can use IPython with Python2 or Python3 which depends on which python you set in zeppelin.python
.
pip install jupyter pip install grpcio pip install protobuf
zeppelin.python
points to the python under anaconda)pip install grpcio pip install protobuf
Zeppelin will check the above prerequisites when using %python
, if IPython prerequisites are met, %python
would use IPython interpreter, otherwise it would use vanilla Python interpreter in %python
.
In addition to all the basic functions of the vanilla python interpreter, you can use all the IPython advanced features as you use it in Jupyter Notebook. Take a look at tutorial note Python Tutorial/1. IPython Basic
and Python Tutorial/2. IPython Visualization Tutorial
for how to use IPython in Zeppelin.
%python.ipython #python help range? #timeit %timeit range(100)
%python.ipython %matplotlib inline import matplotlib.pyplot as plt print("hello world") data=[1,2,3,4] plt.figure() plt.plot(data)
%python.ipython !pip install pandas
e.g. You can use hvplot in the same way as in Jupyter, Take a look at tutorial note Python Tutorial/2. IPython Visualization Tutorial
for more visualization examples.
Type tab
can give you all the completion candidates just like in Jupyter.
Apache Zeppelin Table Display System provides built-in data visualization capabilities. Python interpreter leverages it to visualize Pandas DataFrames via z.show()
API.
For example:
By default, z.show
only display 1000 rows, you can configure zeppelin.python.maxResult
to adjust the max number of rows.
There is a convenience %python.sql
interpreter that matches Apache Spark experience in Zeppelin and enables usage of SQL language to query Pandas DataFrames and visualization of results through built-in Table Display System. %python.sql
can access dataframes defined in %python
.
Prerequisites
pip install pandas
pip install -U pandasql
Here's one example:
first paragraph
%python import pandas as pd rates = pd.read_csv(“bank.csv”, sep=“;”)
* next paragraph ```sql %python.sql SELECT * FROM rates WHERE age < 40
You can leverage Zeppelin Dynamic Form inside your Python code.
Example :
%python ### Input form print(z.input("f1","defaultValue")) ### Select form print(z.select("f2",[("o1","1"),("o2","2")],"o1")) ### Checkbox form print("".join(z.checkbox("f3", [("o1","1"), ("o2","2")],["o1"])))
Python interpreter create a variable z
which represent ZeppelinContext
for you. User can use it to do more fancy and complex things in Zeppelin.
Zeppelin supports to run interpreter in yarn cluster which means the python interpreter can run in a yarn container. This can achieve better multi-tenant for python interpreter especially when you already have a hadoop yarn cluster.
But there's one critical problem to run python in yarn cluster: how to manage the python environment in yarn container. Because hadoop yarn cluster is a distributed cluster environment which is composed of many nodes, and your python interpreter can start in any node. It is not practical to manage python environment in each node beforehand.
So in order to run python in yarn cluster, we would suggest you to use conda to manage your python environment, and Zeppelin can ship your conda environment to yarn container, so that each python interpreter can have its own python environment without affecting each other.
Python interpreter in yarn cluster only works for IPython, so make sure IPython's prerequisites are met. So make sure including the following packages in Step 1.
We would suggest you to use conda-pack to create archive of conda environment, and ship it to yarn container. Otherwise python interpreter will use the python executable file in PATH of yarn container.
Here's one example of yaml file which could be used to create a conda environment with python 3 and some useful python libraries.
python_3_env.yml
name: python_3_env channels: - conda-forge - defaults dependencies: - python=3.7 - jupyter - grpcio - protobuf - pycodestyle - numpy - pandas - scipy - pandasql - panel - pyyaml - seaborn - plotnine - hvplot - intake - intake-parquet - intake-xarray - altair - vega_datasets - pyarrow
conda env create -f python_3_env.yml
mamba env create -f python_3_env
conda
conda pack -n python_3_env
Specify the following properties to enable yarn mode for python interpreter.
%python.conf zeppelin.interpreter.launcher yarn zeppelin.yarn.dist.archives /home/hadoop/python_3_env.tar.gz#environment zeppelin.interpreter.conda.env.name environment
Setting zeppelin.interpreter.launcher
as yarn
will launch python interpreter in yarn cluster.
zeppelin.yarn.dist.archives
is the python conda environment tar which is created in step 1. This tar will be shipped to yarn container and untar in the working directory of yarn container. environment
in /home/hadoop/python_3.tar.gz#environment
is the folder name after untar.
This folder name should be the same as zeppelin.interpreter.conda.env.name
. Usually we name it as environment
here.
By default, PythonInterpreter will use python command defined in zeppelin.python
property to run python process. The interpreter can use all modules already installed (with pip, easy_install...)
Conda is an package management system and environment management system for python. %python.conda
interpreter lets you change between environments.
get the Conda Information:
%python.conda info
list the Conda environments:
%python.conda env list
create a conda enviornment:
%python.conda create --name [ENV NAME]
activate an environment (python interpreter will be restarted):
%python.conda activate [ENV NAME]
deactivate
%python.conda deactivate
get installed package list inside the current environment
%python.conda list
install package
%python.conda install [PACKAGE NAME]
uninstall package
%python.conda uninstall [PACKAGE NAME]
%python.docker
interpreter allows PythonInterpreter creates python process in a specified docker container.
activate an environment
%python.docker activate [Repository] %python.docker activate [Repository:Tag] %python.docker activate [Image Id]
deactivate
%python.docker deactivate
# activate latest tensorflow image as a python environment %python.docker activate gcr.io/tensorflow/tensorflow:latest
Join our community to discuss with others.