This documentation is to guide SINGA developers to setup Jenkins service to support continuous integration on GPU systems. After each commit,
Continuous integration for CPU systems is enabled via Travis.
Jenkins Official Wiki The slave nodes for running different building environments are configured under ‘Manage Jenkins’->‘Manage nodes’.
Create a multi-configuration project and configure project as follows:
This job automatically pulls latest commits from Apache incubator-singa github repository, then for different environments
The working nodes (or Docker containers) are configured in Jenkins-Manage Jenkins-Mange Nodes. Each node should configure the following environment variable
https://github.com/apache/incubator-singa
https://github.com/apache/incubator-singa
*/master
H/30 * * * *
(pull every 30 minutes)lang
values CUDA
(Add CPP if you want to test CPU code)env
Node/label: tick available nodescompile and do unit test on GPU Execute shell - command - bash -ex tool/jenkins/test.sh $lang
$lang
is set in *Configuration Matrix section
create Debian package Execute shell - command - bash -ex tool/debian/build.sh --python --$lang
create conda package Execute shell - command -
git push https://username:token@github.com/nusdbsystem/incubator-singa.git -f bash -ex tool/jenkins/jenkins_test.sh $lang export CONDA_BLD_PATH=/root/conda-bld-$BUILD_NUMBER mkdir $CONDA_BLD_PATH /root/miniconda/bin/conda-build tool/conda /root/miniconda/bin/anaconda -t ANACONDA_UPLOAD_TOKEN upload -u nusdbsystem -l main $CONDA_BLD_PATH/linux-64/singa-*.tar.bz2 --force
It first pushes to a mirror site to invoke travis-ci for CPU package creation; Then it compiles and runs unit tests; Finally it creates the conda package for GPU and upload it.
**/gtest.xml, **/unittest.xml
build/debian/**.deb
../debian/build.sh packages the .deb file into $BUILD_ID.tar.gz. Inside the tar file, the folder layout is build_id/commit_hash/os_lang/*.deb
, where os_lang
is the combination of os version, device programming language (cuda/cpp/opencl) and cudnn version.
In Manage Jenkins
-Configure System
, configure the SSH for connecting to the remote public server and set the target folder location
Source files - build/debian/*.tar.gz
Remove prefix - `build/debian
Remote directory - debian
Exec a command on the remote server to decompress the package and add a symlink to the latest build. E.g., on a Solaris server the command is
cd <target_folder>/debian && gunzip $BUILD_ID.tar.gz && tar xf $BUILD_ID.tar && chmod -R 755 $BUILD_ID && /bin/rm -f $BUILD_ID.tar && /bin/rm -f latest && ln -s $BUILD_ID/* latest
The file links on the remote public server would be like
debian/32/84d56b7/ubuntu14.04-cpp/singa-1.0.1.deb debian/32/84d56b7/ubuntu14.04-cuda8.0-cudnn5/singa-1.0.1.deb
We provide in a number of singa docker images for Jenkins to use as slaves. To run the docker images,
nvidia-docker run --name <jenkins-slaveXX> -d <Image ID>
This job is triggered upon any changes to the files of the doc/
folder. It does the following tasks,
The Jenkins job configuration is similar as above except the following fields,
doc/*
bash -ex tool/jenkins/jenkins_doc.sh
Post-build Actions
The Docker image for the Jenkins slave node is at docker/ubuntu16.04/runtime/Dockerfile
. To build the docker image,
# under the docker/ubuntu16.04/runtime/ folder $ docker built -t singa:doc .
To start the slave node
$ docker run --name singa-doc -d singa:doc $ docker exec -it singa-doc /bin/bash $ svn co https://svn.apache.org/repos/asf/incubator/singa/site/trunk # update ~/.subversion/config to set 'store-password=yes' # to set password free commit, we have to do a manual commit at first. # change any file (add spaces) inside trunk/ to commit a message $ svn commit -m "test" --username <committer id> --password <passwd>
Use Role Strategy Plugin
to give read access for anonymous users.