This documentation is to guide Singa developers to setup Jenkins service for continuous integration of Singa. After each commit,
Continuous integration for CPU systems is enabled via Travis. Hence, Jenkins is mainly used for CI on GPUs.
Jenkins Official Wiki The slave nodes for running different building environments are configured under ‘Manage Jenkins’->‘Manage nodes’.
Change Jenkins time zone by executing the following code in ‘Mange jenkins’ -> ‘Script Console’:
System.setProperty('org.apache.commons.jelly.tags.fmt.timeZone', 'Asia/Singapore')
Create a multi-configuration project and configure project as follows:
This job automatically pulls latest commits from Apache incubator-singa github repository, then for different environments
https://github.com/apache/incubator-singa
https://github.com/apache/incubator-singa
*/master
H/30 * * * *
(pull every 30 minutes)lang
values CUDA
(Add CPP if you want to test CPU code)env
Node/label: tick available nodesThe building script can do the following tasks:
compile and do unit test on GPU Execute shell - command - bash -ex tool/jenkins/test.sh
update another github repo with the new commits to invoke travis (for cpu test and conda package generation) Execute shell - command - git push https://<username:token>@github.com/nusdbsystem/incubator-singa.git -f
create conda package and upload it to anaconda cloud Execute shell - command
/root/miniconda/bin/conda-build tool/conda/singa /root/miniconda/bin/anaconda -t <ANACONDA_UPLOAD_TOKEN> upload -u nusdbsystem -l main /root/miniconda/linux-64/singa-*.so.*.tar.bz2 --force
(optional) create Debian package Execute shell - command - bash -ex tool/debian/build.sh --python --$lang
**/gtest.xml, **/unittest.xml
build/debian/**.deb
../debian/build.sh packages the .deb file into $BUILD_ID.tar.gz. Inside the tar file, the folder layout is build_id/commit_hash/os_lang/*.deb
, where os_lang
is the combination of os version, device programming language (cuda/cpp/opencl) and cudnn version.
In Manage Jenkins
-Configure System
, configure the SSH for connecting to the remote public server and set the target folder location
Source files - build/debian/*.tar.gz
Remove prefix - `build/debian
Remote directory - debian
Exec a command on the remote server to decompress the package and add a symlink to the latest build. E.g., on a Solaris server the command is
cd <target_folder>/debian && gunzip $BUILD_ID.tar.gz && tar xf $BUILD_ID.tar && chmod -R 755 $BUILD_ID && /bin/rm -f $BUILD_ID.tar && /bin/rm -f latest && ln -s $BUILD_ID/* latest
The file links on the remote public server would be like
debian/32/84d56b7/ubuntu14.04-cpp/singa-1.0.1.deb debian/32/84d56b7/ubuntu14.04-cuda8.0-cudnn5/singa-1.0.1.deb
We provide different Singa Dockerfiles for Jenkins to use as working nodes.
To run the docker images,
nvidia-docker run --name <node name> -P -d <Image ID>
To add the container into a network for easy access
docker network create <network name> docker network connect <network name> <node name>
After connecting both the jenkins and node contaniners into the same network, we can ssh to the node from jenkins container like
# inside jenkins container ssh root@<node name>
You need execute the above command manually for the first ssh login.
In the Jenkins node configuration page, the container name is used to configure the Host
field. Notice that Oracle username and account are required to luanch the node by Jenkins.
The working nodes (or Docker containers) are configured in Jenkins-Manage Jenkins-Mange Nodes. Each node should configure the following environment variable
export CUDA=<cuda version, e.g., 9.0>
Dockerfiles are provided to create the working nodes.
This job is triggered upon any changes to the files of the doc/
folder. It does the following tasks,
The Jenkins job configuration is similar as above except the following fields,
Source Code Management - Git - Additional Behaviors - Include Region doc/*
Build - Execute Shell - Command
bash -ex tool/jenkins/gen_doc.sh
No Post-build Actions
The docker images used for testing also be used for document generation. We have to manually configure something inside the docker container. First, we start the container
$docker run --name singa-doc -d <docker image> # docker network connect jenkins singa-doc $ docker exec -it singa-doc /bin/bash
Next, we do the first commit to the svn repo.
$ svn co https://svn.apache.org/repos/asf/incubator/singa/site/trunk # update ~/.subversion/config to set 'store-password=yes' # to set password free commit, we have to do a manual commit at first. # change any file (add spaces) inside trunk/ to commit a message $ svn commit -m "test" --username <committer id> --password <passwd>
Use Role Strategy Plugin
to give read access for anonymous users.