Nightly Tests for MXNet

These are some longer running tests that are scheduled to run every night, for master and for latest release branches.

There are two Jenkins pipelines that run these tests -

Add your test script to the MXNet repo's tests/nightly/ folder. Make sure to describe in a readme or in the comments the purpose of the test.

Your test must run on the CI slaves only within an official docker container available at ci/docker
Make sure your script itself does not install anything on the slaves. All dependencies must be added to the dockerfile.
For most tests you should be able to find an existing dockerfile that contains all the necessary dependencies - select based on the required os, cpu vs gpu, necessary dependencies etc
However, If there is no dockerfile which installs the required dependencies do the following - a. Add your dependencies to a new install script in ci/docker/install b. Now either run this install script within an existing dockerfile or create a new dockerfile.

If the test runs on the MXNet source, modify tests/nightly/Jenkinsfile -

Add a function to ci/docker/runtimefunctions.sh which calls your test script.
In the Jenkinsfile, add a new test similar to existing tests under the NightlyTests stage. Make sure to call the right runtime function

If the test runs on MXNet binaries modify tests/nightly/JenkinsfileForBinaries -

Add a function to ci/docker/runtimefunctions.sh which calls your test script.
In the Jenkinsfile, if needed, add a new build similar to existing tests under the Build stage.
Add a new test similar to existing tests under the NightlyTests stage. Make sure to call the right runtime function