| <!-- |
| Licensed to the Apache Software Foundation (ASF) under one |
| or more contributor license agreements. See the NOTICE file |
| distributed with this work for additional information |
| regarding copyright ownership. The ASF licenses this file |
| to you under the Apache License, Version 2.0 (the |
| "License"); you may not use this file except in compliance |
| with the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, |
| software distributed under the License is distributed on an |
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| KIND, either express or implied. See the License for the |
| specific language governing permissions and limitations |
| under the License. |
| --> |
| |
| # Build SINGA from Source |
| |
| |
| The source files could be downloaded either as a |
| [tar.gz file](https://dist.apache.org/repos/dist/dev/singa/), or as a git repo |
| |
| $ git clone https://github.com/apache/singa.git |
| $ cd singa/ |
| |
| If you want to contribute code to SINGA, refer to [this page]() for the steps and requirements. |
| |
| ## Use Conda to build SINGA |
| |
| Conda-build is a building tool that installs the dependent libraries from anaconda cloud and |
| executes the building scripts. |
| |
| To install conda-build (after installing conda) |
| |
| conda install conda-build |
| |
| ### Build CPU Version |
| |
| To build the CPU version of SINGA |
| |
| conda build tool/conda/singa/ |
| |
| The above commands have been tested on Ubuntu (14.04, 16.04 and 18.04) and macOS 10.11. |
| Refer to the [Travis-CI page](https://travis-ci.org/apache/singa) for more information. |
| |
| ### Build GPU Version |
| |
| To build the GPU version of SINGA, the building machine must have Nvida GPU, and the CUDA driver (>= 384.81), CUDA toolkit (>=9) and cuDNN (>=7) must have be installed. The following two Docker images provide the building environment: |
| |
| 1. apache/singa:conda-cuda9.0 |
| 2. apache/singa:conda-cuda10.0 |
| |
| Once the building environment is ready, you need to export the CUDA version first, and then run conda command to build SINGA |
| |
| export CUDA=x.y (e.g. 9.0) |
| conda build tool/conda/singa/ |
| |
| ### Post Processing |
| |
| The location of the generated package file (`.tar.gz`) is shown on the screen. |
| The generated package can be installed directly, |
| |
| conda install -c conda-forge --use-local <path to the package file> |
| |
| or uploaded to anaconda cloud for others to download and install. You need to register an account on anaconda for [uploading the package](https://docs.anaconda.com/anaconda-cloud/user-guide/getting-started/). |
| |
| conda install anaconda-client |
| anaconda login |
| anaconda upload -l main <path to the package file> |
| |
| After uploading the package to the cloud, you can see it on [Anaconda Cloud](https://anaconda.org/) website or via the following command |
| |
| conda search -c <anaconda username> singa |
| |
| Each specific SINGA package is identified by the version and build string. To install a specific SINGA package, you need to provide all the information, e.g., |
| |
| conda install -c <anaconda username> -c conda-forge singa=2.1.0.dev=cpu_py36 |
| |
| To make the installation command simple, you can create the following additional packages which depend on the latest CPU and GPU SINGA packages. |
| |
| # for singa-cpu |
| conda build tool/conda/cpu/ --python=3.6 |
| conda build tool/conda/cpu/ --python=3.7 |
| # for singa-gpu |
| conda build tool/conda/gpu/ --python=3.6 |
| conda build tool/conda/gpu/ --python=3.7 |
| |
| Therefore, when you run |
| |
| conda install -c <anaconda username> -c conda-forge singa-xpu |
| |
| (`xpu` is either 'cpu' or 'gpu'), the corresponding real SINGA package is installed as the dependent library. |
| |
| ## Use native tools to build SINGA on Ubuntu |
| |
| Refer to SINGA [Dockerfiles](https://github.com/apache/singa/blob/master/tool/docker/devel/ubuntu/cuda9/Dockerfile#L30) |
| for the instructions of installing the dependent libraries on Ubuntu 16.04. You can also create a Docker container using the [devel images]() and build SINGA inside the container. |
| To build SINGA with GPU, MKLDNN, Python and unit tests, run the following instructions |
| |
| mkdir build # at the root of singa folder |
| cd build |
| cmake -DENABLE_TEST=ON -DUSE_CUDA=ON -DUSE_MKLDNN=ON -DUSE_PYTHON3=ON .. |
| make |
| cd python |
| pip install . |
| |
| The details of the CMake options are explained in the last section of this page. |
| The last command install the Python package. You can also run `pip install -e .`, which creates symlinks instead of copying the Python files into the site-package folder. |
| |
| If SINGA is compiled with ENABLE_TEST=ON, you can run the unit tests by |
| |
| $ ./bin/test_singa |
| |
| You can see all the testing cases with testing results. If SINGA passes all |
| tests, then you have successfully installed SINGA. |
| |
| |
| ## Use native tools to Build SINGA on Centos7 |
| |
| Building from source will be different for Centos7 as package names differ.Follow the instructions given below. |
| |
| ### Installing dependencies |
| |
| Basic packages/libraries |
| |
| sudo yum install freetype-devel libXft-devel ncurses-devel openblas-devel blas-devel lapack devel atlas-devel kernel-headers unzip wget pkgconfig zip zlib-devel libcurl-devel cmake curl unzip dh-autoreconf git python-devel glog-devel protobuf-devel |
| For build-essential |
| |
| sudo yum group install "Development Tools" |
| For installing swig |
| |
| sudo yum install pcre-devel |
| wget http://prdownloads.sourceforge.net/swig/swig-3.0.10.tar.gz |
| tar xvzf swig-3.0.10.tar.gz |
| cd swig-3.0.10.tar.gz |
| ./configure --prefix=${RUN} |
| make |
| make install |
| For installing gfortran |
| |
| sudo yum install centos-release-scl-rh |
| sudo yum --enablerepo=centos-sclo-rh-testing install devtoolset-7-gcc-gfortran |
| For installing pip and other packages |
| |
| sudo yum install epel-release |
| sudo yum install python-pip |
| pip install matplotlib numpy pandas scikit-learn pydot |
| |
| ### Installation |
| Follow steps 1-5 of _Use native tools to build SINGA on Ubuntu_ |
| |
| ### Testing |
| You can run the unit tests by, |
| |
| $ ./bin/test_singa |
| You can see all the testing cases with testing results. If SINGA passes all |
| tests, then you have successfully installed SINGA. |
| |
| ## Compile SINGA on Windows |
| |
| Instructions for building on Windows with Python support can be found [here](install_win.html). |
| |
| ## More details about the compilation options |
| |
| ### USE_MODULES (deprecated) |
| |
| If protobuf and openblas are not installed, you can compile SINGA together with them |
| |
| $ In SINGA ROOT folder |
| $ mkdir build |
| $ cd build |
| $ cmake -DUSE_MODULES=ON .. |
| $ make |
| |
| cmake would download OpenBlas and Protobuf (2.6.1) and compile them together |
| with SINGA. |
| |
| You can use `ccmake ..` to configure the compilation options. |
| If some dependent libraries are not in the system default paths, you need to export |
| the following environment variables |
| |
| export CMAKE_INCLUDE_PATH=<path to the header file folder> |
| export CMAKE_LIBRARY_PATH=<path to the lib file folder> |
| |
| ### USE_PYTHON |
| |
| Option for compiling the Python wrapper for SINGA, |
| |
| $ cmake -DUSE_PYTHON=ON .. |
| $ make |
| $ cd python |
| $ pip install . |
| |
| |
| ### USE_CUDA |
| |
| Users are encouraged to install the CUDA and |
| [cuDNN](https://developer.nvidia.com/cudnn) for running SINGA on GPUs to |
| get better performance. |
| |
| SINGA has been tested over CUDA 9/10, and cuDNN 7. If cuDNN is |
| installed into non-system folder, e.g. /home/bob/local/cudnn/, the following |
| commands should be executed for cmake and the runtime to find it |
| |
| $ export CMAKE_INCLUDE_PATH=/home/bob/local/cudnn/include:$CMAKE_INCLUDE_PATH |
| $ export CMAKE_LIBRARY_PATH=/home/bob/local/cudnn/lib64:$CMAKE_LIBRARY_PATH |
| $ export LD_LIBRARY_PATH=/home/bob/local/cudnn/lib64:$LD_LIBRARY_PATH |
| |
| The cmake options for CUDA and cuDNN should be switched on |
| |
| # Dependent libs are install already |
| $ cmake -DUSE_CUDA=ON .. |
| $ make |
| |
| ### USE_MKLDNN |
| |
| User can enable MKL-DNN to enhance the performance of CPU computation. |
| |
| Installation guide of MKL-DNN could be found [here](https://github.com/intel/mkl-dnn#installation). |
| |
| SINGA has been tested over MKL-DNN v0.17.2. |
| |
| To build SINGA with MKL-DNN support: |
| |
| # Dependent libs are installed already |
| $ cmake -DUSE_MKLDNN=ON .. |
| $ make |
| |
| |
| |
| ### USE_OPENCL |
| |
| SINGA uses opencl-headers and viennacl (version 1.7.1 or newer) for OpenCL support, which |
| can be installed using via |
| |
| # On Ubuntu 16.04 |
| $ sudo apt-get install opencl-headers, libviennacl-dev |
| # On Fedora |
| $ sudo yum install opencl-headers, viennacl |
| |
| Additionally, you will need the OpenCL Installable Client Driver (ICD) for the platforms that you want to run OpenCL on. |
| |
| * For AMD and nVidia GPUs, the driver package should also install the correct OpenCL ICD. |
| * For Intel CPUs and/or GPUs, get the driver from the [Intel website.](https://software.intel.com/en-us/articles/opencl-drivers) Note that the drivers provided on that website only supports recent CPUs and Iris GPUs. |
| * For older Intel CPUs, you can use the `beignet-opencl-icd` package. |
| |
| Note that running OpenCL on CPUs is not currently recommended because it is slow. |
| Memory transfer is on the order of whole seconds (1000's of ms on CPUs as compared to 1's of ms on GPUs). |
| |
| More information on setting up a working OpenCL environment may be found [here](https://wiki.tiker.net/OpenCLHowTo). |
| |
| If the package version of ViennaCL is not at least 1.7.1, you will need to build it from source: |
| |
| Clone [the repository from here](https://github.com/viennacl/viennacl-dev), checkout the `release-1.7.1` tag and build it. |
| Remember to add its directory to `PATH` and the built libraries to `LD_LIBRARY_PATH`. |
| |
| To build SINGA with OpenCL support (tested on SINGA 1.1): |
| |
| $ cmake -DUSE_OPENCL=ON .. |
| $ make |
| |
| |
| ### PACKAGE |
| |
| This setting is used to build the Debian package. Set PACKAGE=ON and build the package with make command like this: |
| |
| $ cmake -DPACKAGE=ON |
| $ make package |
| |
| ## FAQ |
| |
| * Q: Error from 'import singa' |
| |
| A: Please check the detailed error from `python -c "from singa import _singa_wrap"`. Sometimes it is caused by the dependent libraries, e.g. there are multiple versions of protobuf, missing of cudnn, numpy version mismatch. Following steps show the solutions for different cases |
| 1. Check the cudnn and cuda. If cudnn is missing or not match with the wheel version, you can download the correct version of cudnn into ~/local/cudnn/ and |
| |
| $ echo "export LD_LIBRARY_PATH=/home/<yourname>/local/cudnn/lib64:$LD_LIBRARY_PATH" >> ~/.bashrc |
| |
| 2. If it is the problem related to protobuf. You can install protobuf (3.6.1) from source into a local folder, say ~/local/; Decompress the tar file, and then |
| |
| $ ./configure --prefix=/home/<yourname>local |
| $ make && make install |
| $ echo "export LD_LIBRARY_PATH=/home/<yourname>/local/lib:$LD_LIBRARY_PATH" >> ~/.bashrc |
| $ source ~/.bashrc |
| |
| 3. If it cannot find other libs including python, then create virtual env using pip or conda; |
| |
| 4. If it is not caused by the above reasons, go to the folder of `_singa_wrap.so`, |
| |
| $ python |
| >> import importlib |
| >> importlib.import_module('_singa_wrap') |
| |
| Check the error message. For example, if the numpy version mismatches, the error message would be, |
| |
| RuntimeError: module compiled against API version 0xb but this version of numpy is 0xa |
| |
| Then you need to upgrade the numpy. |
| |
| |
| * Q: Error from running `cmake ..`, which cannot find the dependent libraries. |
| |
| A: If you haven't installed the libraries, install them. If you installed |
| the libraries in a folder that is outside of the system folder, e.g. /usr/local, |
| you need to export the following variables |
| |
| $ export CMAKE_INCLUDE_PATH=<path to your header file folder> |
| $ export CMAKE_LIBRARY_PATH=<path to your lib file folder> |
| |
| |
| * Q: Error from `make`, e.g. the linking phase |
| |
| A: If your libraries are in other folders than system default paths, you need |
| to export the following varaibles |
| |
| $ export LIBRARY_PATH=<path to your lib file folder> |
| $ export LD_LIBRARY_PATH=<path to your lib file folder> |
| |
| |
| * Q: Error from header files, e.g. 'cblas.h no such file or directory exists' |
| |
| A: You need to include the folder of the cblas.h into CPLUS_INCLUDE_PATH, |
| e.g., |
| |
| $ export CPLUS_INCLUDE_PATH=/opt/OpenBLAS/include:$CPLUS_INCLUDE_PATH |
| |
| * Q:While compiling SINGA, I get error `SSE2 instruction set not enabled` |
| |
| A:You can try following command: |
| |
| $ make CFLAGS='-msse2' CXXFLAGS='-msse2' |
| |
| * Q:I get `ImportError: cannot import name enum_type_wrapper` from google.protobuf.internal when I try to import .py files. |
| |
| A: You need to install the python binding of protobuf, which could be installed via |
| |
| $ sudo apt-get install protobuf |
| |
| or from source |
| |
| $ cd /PROTOBUF/SOURCE/FOLDER |
| $ cd python |
| $ python setup.py build |
| $ python setup.py install |
| |
| * Q: When I build OpenBLAS from source, I am told that I need a Fortran compiler. |
| |
| A: You can compile OpenBLAS by |
| |
| $ make ONLY_CBLAS=1 |
| |
| or install it using |
| |
| $ sudo apt-get install libopenblas-dev |
| |
| * Q: When I build protocol buffer, it reports that GLIBC++_3.4.20 not found in /usr/lib64/libstdc++.so.6. |
| |
| A: This means the linker found libstdc++.so.6 but that library |
| belongs to an older version of GCC than was used to compile and link the |
| program. The program depends on code defined in |
| the newer libstdc++ that belongs to the newer version of GCC, so the linker |
| must be told how to find the newer libstdc++ shared library. |
| The simplest way to fix this is to find the correct libstdc++ and export it to |
| LD_LIBRARY_PATH. For example, if GLIBC++_3.4.20 is listed in the output of the |
| following command, |
| |
| $ strings /usr/local/lib64/libstdc++.so.6|grep GLIBC++ |
| |
| then you just set your environment variable as |
| |
| $ export LD_LIBRARY_PATH=/usr/local/lib64:$LD_LIBRARY_PATH |
| |
| * Q: When I build glog, it reports that "src/logging_unittest.cc:83:20: error: ‘gflags’ is not a namespace-name" |
| |
| A: It maybe that you have installed gflags with a different namespace such as "google". so glog can't find 'gflags' namespace. |
| Because it is not necessary to have gflags to build glog. So you can change the configure.ac file to ignore gflags. |
| |
| 1. cd to glog src directory |
| 2. change line 125 of configure.ac to "AC_CHECK_LIB(gflags, main, ac_cv_have_libgflags=0, ac_cv_have_libgflags=0)" |
| 3. autoreconf |
| |
| After this, you can build glog again. |
| |
| * Q: When using virtual environment, every time I run pip install, it would reinstall numpy. However, the numpy would not be used when I `import numpy` |
| |
| A: It could be caused by the `PYTHONPATH` which should be set to empty when you are using virtual environment to avoid the conflicts with the path of |
| the virtual environment. |
| |
| * Q: When compiling PySINGA from source, there is a compilation error due to the missing of <numpy/objectarray.h> |
| |
| A: Please install numpy and export the path of numpy header files as |
| |
| $ export CPLUS_INCLUDE_PATH=`python -c "import numpy; print numpy.get_include()"`:$CPLUS_INCLUDE_PATH |
| |
| * Q: When I run SINGA in Mac OS X, I got the error "Fatal Python error: PyThreadState_Get: no current thread Abort trap: 6" |
| |
| A: This error happens typically when you have multiple version of Python on your system and you installed SINGA via pip (this problem is resolved for installation via conda), |
| e.g, the one comes with the OS and the one installed by Homebrew. The Python linked by PySINGA must be the same as the Python interpreter. |
| You can check your interpreter by `which python` and check the Python linked by PySINGA via `otool -L <path to _singa_wrap.so>`. |
| To fix this error, compile SINGA with the correct version of Python. |
| In particular, if you build PySINGA from source, you need to specify the paths when invoking [cmake](http://stackoverflow.com/questions/15291500/i-have-2-versions-of-python-installed-but-cmake-is-using-older-version-how-do) |
| |
| $ cmake -DPYTHON_LIBRARY=`python-config --prefix`/lib/libpython2.7.dylib -DPYTHON_INCLUDE_DIR=`python-config --prefix`/include/python2.7/ .. |
| |
| If installed PySINGA from binary packages, e.g. debian or wheel, then you need to change the python interpreter, e.g., reset the $PATH to put the correct path of Python at the front position. |