| % |
| % Licensed to the Apache Software Foundation (ASF) under one |
| % or more contributor license agreements. See the NOTICE file |
| % distributed with this work for additional information |
| % regarding copyright ownership. The ASF licenses this file |
| % to you under the Apache License, Version 2.0 (the |
| % "License"); you may not use this file except in compliance |
| % with the License. You may obtain a copy of the License at |
| % |
| % http://www.apache.org/licenses/LICENSE-2.0 |
| % |
| % Unless required by applicable law or agreed to in writing, |
| % software distributed under the License is distributed on an |
| % "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| % KIND, either express or implied. See the License for the |
| % specific language governing permissions and limitations |
| % under the License. |
| % |
| % Create well-known link to this spot for HTML version |
| \ifpdf |
| \else |
| \HCode{<a name='DUCC\_SIM'></a>} |
| \fi |
| \chapter{Simulation and System Testing} |
| \label{chap:simulation} |
| This chapter describes the large-scale testing and cluster-simulation |
| tools supplied with DUCC. This is of use mostly to contributors and |
| developers of DUCC itself. |
| |
| DUCC is shipped with support for simulating large clusters of arbitrarily |
| configured nodes. A simple control file describes some number of |
| simulated nodes of arbitrary memory sizes. DUCC's design allows multiples |
| of these to be spawned on a single node, or on a small set of nodes with |
| multiple simulated nodes apiece. The standard testing configuration |
| used for most of the development of DUCC consisted of four |
| physical 32-GB machines running 52 simulated nodes of varying memory |
| sizes from 32 to 128-GB each. |
| |
| To simulate job loads, a simple UIMA-AS job that sleeps for some easily configured |
| length of time was constructed. Another control file is used to |
| generate \hyperref[sec:cli.ducc-submit]{job specifications} requesting randomly-chosen |
| job parameters such as memory requirements, service dependencies, scheduling classes, and so on. |
| |
| The test suite contains a simple UIMA Analysis Engine called |
| {\tt FixedSleepAE}, and a simple Collection Reader called |
| {\tt FixedSleepCR}. The CR reads a set of sleep times, creates |
| CASs, and ships them to the AEs via DUCC's Job Driver. The CAS |
| contains the time to sleep and various parameters regarding |
| error injection. |
| |
| The AE receives a CAS, performs error injection if requested, and |
| sleeps the indicated period of time, simulating actual computation |
| but requiring very few physical resources. Hence, many of these |
| may be run simultaneously on relatively modest hardware. |
| |
| Developers may construct arbitrary jobs by creating a file with |
| sleep times designed to exercise what ever is necessary. DUCC |
| ships with the three primary job collections (test suites) used |
| during initial development. The suites are based on actual |
| workloads and have shown to be very robust for proving the correctness |
| of the DUCC code under stress. |
| |
| The cluster simulator has been also been run on a 4GB iMac with 8 simulated Agents, an 8GB MacBook with |
| the same configuration, a 32GB iMac with up to 40 simulated Agents. It has also been scaled |
| up to run on 8 45GB Intel nodes running Linux, simulating 20TB of memory. |
| |
| The rest of this chapter describes the mechanics of using these tools. |
| |
| \section{Cluster Simulation} |
| |
| \subsection{Overview} |
| Cluster-based tools such as DUCC are very hard to test and debug |
| because all interesting problems occur only when the system is |
| under stress. Acquisition of a cluster of sufficient size to |
| expose the interesting problems is usually not practical. |
| |
| DUCC's design divorces all the DUCC processes from specific IP |
| addresses or node names. ActiveMQ is used as a nameserver and |
| packet router so that all messages can be delivered by name, |
| irrespective of the physical hardware the destination process |
| may reside upon. |
| |
| A DUCC system is comprised of three types of processes (daemons): |
| \begin{enumerate} |
| \item The DUCC management daemons: |
| \begin{itemize} |
| \item The Orchestrator (OR). This is the primary point of |
| entry to the system and is responsible for managing |
| the life cycle of all work in the system. |
| \item The Process Manager (PM). This is responsible for |
| managing message flow to and from the DUCC Agents. |
| \item The \hyperref[chap:rm]{Resource Manager} (RM). This is responsible for |
| apportioning system resources among submitted work |
| (jobs, reservations, services). |
| \item The \hyperref[chap:services]{Service Manager} (SM). This is responsible for |
| keeping services active and available as needed. |
| \item The Web Server (WS). This process listens to all |
| the state messages in the system to provide a coherent |
| view of DUCC to the outside world. |
| \end{itemize} |
| \item The DUCC Node Agents, or simply, Agents. There is |
| one Agent running on every physical node. |
| \item The ActiveMQ Broker. All message flow in the system |
| is directed through the ActiveMQ broker, with the exception |
| of the CLI, (which uses HTTP). |
| \end{enumerate} |
| |
| Normally, the DUCC Agents report the name, IP address, and physical memory of the node |
| they actually do reside upon. This is simply for convenience. |
| It is possible to parametrize the DUCC Agents to report any arbitrary |
| name and address to the DUCC. DUCC components that need to know |
| about Node Agents establish subscriptions to the Agent publications |
| with ActiveMQ and build up their internal structures from the |
| node identities in the Agent publications. Processes which normally |
| establish agent listeners are are the RM, PM, and WS. |
| |
| It is also possible to parametrize a DUCC agent to cause it to |
| report any arbitrary memory size. Thus, an agent running on a |
| 2GB machine can be started so that it reports 32GB of memory. This |
| parametrization is specifically for testing, of course. |
| |
| The ability to parametrize agent identities and memory sizes is what enables |
| cluster simulation. A control file is used by start-up scripting |
| to spawn multiple agents per node, each with unique identities. |
| |
| \subsection{Node Configuration} |
| |
| A Java properties file is used to configure |
| simulated nodes. There are three types o entries in this file: |
| \begin{description} |
| \item[nodes] This single entry provides the blank-delimited names of the physical nodes |
| participating in the simulated cluster. |
| \item[memory] This single line consists of a blank-delimited set |
| of numbers. Each number corresponds to some memory size, in |
| GB, to be simulated. |
| \item[node descriptions] There are one or more of these. The format |
| of each line is |
| \begin{verbatim} |
| [nodename].[memory] = [count] |
| \end{verbatim} |
| where |
| \begin{description} |
| \item[nodename] is the name of one of the nodes in the {\em nodes} |
| line mentioned above. |
| \item[memory] is one of the memory sizes given in the {\em memory} |
| line mentioned above. |
| \item[count] is the number of simulated agents in the indicated |
| node, with the indicated memory, to be simulated. |
| \end{description} |
| \end{description} |
| |
| For example, the following simulated cluster configuration defines twenty (20) |
| simulated nodes, all to be run on the single physical machine called {\em agentn}. |
| The simulated nodes contain a mix of 31GB, 47GB, and 79GB memory sizes. There |
| are 7 31GB nodes, 7 47GB nodes, and 6 79GB nodes. |
| \begin{verbatim} |
| # names of nodes in the test cluster |
| nodes = agentn |
| |
| # set of memory sizes to configure |
| memory = 31 47 79 |
| |
| # how to configure memories: node.memsize = count |
| agentn.31 = 7 |
| agentn.47 = 7 |
| agentn.79 = 6 |
| \end{verbatim} |
| |
| The nodenames generated by this means are the name of the physical node where |
| the agent is spawned, and a numeric id appended, for example, |
| \begin{verbatim} |
| agentn-1 |
| agentn-2 |
| agentn-3 |
| etc. |
| \end{verbatim} |
| |
| \subsection{Setting up Test Mode} |
| |
| During simulation and testing it is desirable and usually required that DUCC run |
| in unprivileged mode, with all processes belonging to a single userid. Unfortunately, |
| this does not exercise any of the multi-user code paths, especially in the Resource |
| Manager. |
| |
| To accommodate this, DUCC can be configured to run in ``test mode'', such that work |
| is submitted under ``simulated'' userid which DUCC treats as discrete IDs. All actual |
| work is executed under the ownership of the tester however. |
| |
| To establish test mode: |
| \begin{enumerate} |
| \item Ensure that {\em ducc.properties} is configured to point to a non-privileged |
| version of {\em ducc\_ling}. Specifically, configure this line in {\em ducc.properties} |
| \begin{verbatim} |
| ducc.agent.launcher.ducc_spawn_path=/home/ducctest/duccling.dir/amd64/ducc_ling |
| \end{verbatim} |
| in this example a version of {\em ducc\_ling} known to not have elevated privileges |
| it configured. |
| \item Configure test mode in {\em ducc.properties}: |
| \begin{verbatim} |
| ducc.runmode=Test |
| \end{verbatim} |
| IMPORTANT: Do not start DUCC with {\em ducc.runmode=Test} if {\em ducc\_ling} has |
| elevated privileges. Test mode bypasses the authentication and authorization checks |
| that are normally used and the system would run completely open. |
| \end{enumerate} |
| |
| In test mode, jobs may specify what simulated userid is to be used. Most of DUCC does not |
| pay any attention to the user so this works fine, and the parts that do care about the |
| user are bypassed when {\em ducc.runmode=Test} is configured. |
| |
| \subsection{Starting a Simulated Cluster} |
| DUCC provides a start-up script in the directory {\tt \duccruntime/examples/systemtest} |
| called {\tt start\_sim}. |
| |
| WARNING: Cluster simulation is intended for DUCC testing, including error injection. It is |
| similar to flying a high-performance fighter jet. It is intentionally twitchy. Very little |
| checking is done and processes may be started multiple time regardless of whether is sane to |
| do this. |
| |
| To start a simulated cluster, use the {\em start\_sim} script: |
| |
| \paragraph{Description:} |
| The {\em start\_sim} script is used to start a simulated cluster. |
| |
| \paragraph{Usage:} |
| {\em start\_sim} options |
| |
| \paragraph{Options:} |
| \begin{description} |
| \item[-n, --nodelist {[nodelist]}] where the nodelist is a cluster description as |
| described above. |
| \item[-c --components {[component list]}]. The component list is a blank-delimited |
| list of components including {\em or, rm, sm, pm, ws, broker} to start an |
| individual component, or {\em all} to start all of the components. NOTE: It is |
| usually an error to start any of these components more than once. However |
| {\em start\_sim} allows it, to permit error injection. |
| \item[--nothreading] If specified, the command does not run in multi-threaded mode |
| even if it is supported on the local platform. |
| |
| \end{description} |
| |
| \subsection{Stopping a Simulated Cluster} |
| |
| There are two mechanisms for stopping a simulated cluster: |
| \begin{enumerate} |
| \item {\em check\_ducc -k} This looks for all DUCC processes on the nodes in |
| \ducchome/resources/ducc.nodes and issues {\em kill -9} to each process. It |
| then removes the Orchestrator lock file. This is the most violent and |
| surest way to stop a simulated DUCC cluster. In order for this to work, |
| be sure to include the names of all physical nodes used in the simulated cluster |
| in the DUCC configuration file {\em \duccruntime/resources/ducc.nodes.} It |
| is described in the \hyperref[subsec:admin.check-ducc]{administration section} of the book. |
| |
| \item {\em stop\_sim} With no arguments, this attempts to stop all the simulated |
| agents and the management daemons using {\em kill -INT}. It is possible to |
| stop individual agents or management nodes by specifying their component IDs. |
| The kill signals {\em -KILL, -STOP} and {\em -CONT} are all supported. This |
| allows error injection as well as a more orderly shutdown than |
| {\em check\_ducc -k}. |
| \end{enumerate} |
| |
| \begin{sloppypar} |
| Note that \hyperref[subsec:admin.check-ducc]{{\em check\_ducc}} is found in |
| {\em \duccruntime/admin}. The {\em stop\_sim} script is found in {\em |
| duccruntime/examples/systemtest}. |
| \end{sloppypar} |
| |
| The {\em start\_sim} script creates a file called {\em sim.pids} containing the |
| physical node name, Unix process ID (PID), and component ID (ws, sm, or, pm, rm) of |
| each started DUCC component. In the case of agents, each agent is assigned a |
| number as a unique id. These ids are used with {\em stop\_sim} to affect |
| specific processes. If the cluster is stopped without using {\em stop\_sim}, or |
| if it simply crashes, this PID file will get out of date. Fly more carefully |
| next time! |
| |
| {\em stop\_sim} works as follows: |
| \paragraph{Description} |
| The {\em stop\_sim} script is used to stop some or all of a simulated cluster. |
| |
| \paragraph{Usage:} |
| {\em stop\_sim} [options] |
| |
| \paragraph{Options:} |
| \begin{description} |
| \item[-c, --component {[component name]}] where the name is one of {\em |
| rm, sm, pm, or. ws,}. {\em Kill -INT} is used to enable orderly shutdown |
| unless overridden with -k, -p, or -r as described below. |
| \item[-i, --instance {[instance-id]}] where the instance-id is one of the |
| agent ids in ``sim.pids''. {\em Kill -INT} is used to enable orderly shutdown |
| unless overridden with -k, -p, or -r as described below. |
| \item[-k, --kill] Use {\em kill -9} to kill the process. |
| \item[-p, --pause] Signal the process with {\em SIGSTOP}. |
| \item[-r, --resume] Signal the process with {\em SIGCONT}. |
| \item[--nothreading] If specified, the command does not run in multi-threaded mode |
| even if it is supported on the local platform. |
| |
| \end{description} |
| |
| \section{Job Simulation} |
| |
| \subsection{Overview} |
| ``Real'' jobs are highly memory and CPU intensive. For testing and simulation |
| purposes, the jobs need not use anywhere close to their declared memory, and |
| need not consume any CPU at all. The FixedSleepAE is a UIMA analytic that |
| is given a time, in milliseconds, and all it does is sleep for that period |
| of time and then exit. By running many of these in a simulated cluster |
| it is possible to get all the DUCC administrative processes to behave |
| as if there is a real load on the system when in fact all the nodes and |
| jobs are taking minimal resources. |
| |
| The FixedSleepAE is delivered CASs by the FixedSleepCR. This CR reads |
| a standard Java properties file, using the property ``elapsed'' to derive the |
| set of sleep times. On each call to the CR's ``getNext()'' method, the next |
| integer from ``elapsed'' is fetched, packaged into a CAS, and shipped to |
| ActiveMQ where it is picked up by the next available FixedSleepAE. |
| |
| The test driver is given a control file with the names of all the jobs to be |
| submitted in the current run, and the elapsed time to wait between submission |
| of each job. Each job name corresponds to a file that is not an actual |
| DUCC specification, but rather the description of a DUCC specification. Each |
| description is a simple Java properties file. |
| |
| To submit a job, the test driver reads the next job description file |
| derive the number of |
| threads, the simulated user, the desired (simulated) memory for the job, |
| (possibly) the service ID, and the scheduling class for the job. From these |
| it constructs a DUCC \hyperref[sec:cli.ducc-submit]{job specification} and submits it to DUCC. |
| |
| Scripting is used to read the job meta-descriptors and generate a control |
| file that submits the job set with a large set of variations. The same scripting |
| reads each meta-descriptor and modifies it according to the specific parameters |
| of the run, adjust things such as scheduling class, memory size, etc. |
| |
| \subsection{Job meta-descriptors} |
| For each simulated job in a run, a meta-descriptor must be constructed. These may be |
| constructed ``by hand'', or via local scripting, for example from log analysis. (The |
| packaged meta-descriptors are generated from logs of actual workloads.) |
| |
| A meta-descriptor must contain the following properties: |
| \begin{description} |
| \item[tod] This specifies a virtual ``time of day of submission'', starting from time 0, specified |
| in units of milliseconds, when the job is to be submitted. During job generation, this may |
| be used to enforce precise timing of submission of the jobs. |
| \item[elapsed] This is a blank-delimited set of numbers. Each number represents the elapsed time, |
| in milliseconds, for a single work item. There must be one time for each work item. |
| These numbers are placed into CASs by the job's Job Driver and delivered to each Job Process. |
| For example, |
| if this job is to consist of 5 work items of 1, 2, 3, 4 and 5 seconds each, specify |
| \begin{verbatim} |
| elapsed = 1000 2000 3000 4000 5000 |
| \end{verbatim} |
| \item[threads] This is the number of threads per Job Process. It is translated to the |
| {\em process\_thread\_count} parameter in the job specification. |
| \item[user] This is the name of the user who owns the job. It may be any string at |
| all. If DUCC is started in {\em test} mode, this will be shown as the owner of |
| the job in the webserver and the logs. |
| \item[memory] This is the amount of memory to be requested for the job, translating |
| to the job specification's {\em process\_memory\_size} parameter. |
| \item[class] This is the scheduling class for the job. |
| \item[machines] This is the maximum number of processes to be allocated for the |
| job, corresponding to the {\em process\_deployments\_max} parameter. |
| \end{description} |
| |
| For example: |
| \begin{verbatim} |
| tod = 0 |
| elapsed = 253677 344843 349342 392883 276264 560153 162850 744822 431210 91188 840262 843378 |
| threads = 4 |
| user = Rodrigo |
| memory = 20 |
| class = normal |
| machines = 11 |
| \end{verbatim} |
| |
| All the job meta-descriptors for a run must be placed into a single directory. |
| |
| |
| \subsection{{\em Prepare} Descriptors} |
| \label{subsec:simulation.run-description} |
| A {\em prepare descriptor} is also a |
| standard Java properties file. This defines where the set of meta descriptors resides, |
| where to place the modified meta-files, how to assign scheduling classes to the |
| jobs, how to apportion memory sizes, how to apportion services, how long the total |
| run should last, and how to compress sleep times. |
| |
| All parts of the run are randomized, but the randomization can be made deterministic |
| between runs by specifying a seed to the random number generator. |
| |
| Properties include |
| \begin{description} |
| \item[random.seed] This is the random-number generator seed to be used for |
| creating the run. |
| \item[src.dir] This is the directory containing the input-set of meta-specification |
| files. |
| \item[dest.dir] This is the directory that will contain the updated meta-specification |
| files. |
| \item[scheduling.classes] This is a blank-delimited list of the scheduling classes to |
| be randomly assigned to the jobs. |
| \item[scheduling.classes.{[name]}] Here, {\em name} is the name of one of the |
| scheduling classes listed above. The value is a weight, to be used to affect |
| the distribution of scheduling classes among the jobs. |
| \item[job.memory] This is a blank-delimited list of memory sizes to be randomly |
| assigned to each job. |
| \item[job.memory.{[men]}]] Here, {\em mem} is one of the memory sizes specified |
| above. The value is a weight, used to affect the distribution of memory sizes |
| among the jobs. |
| \item[job.services] This is a blank-delimited list of a service id, where the id |
| is one of the services specified in the {\em services.boot} control file. |
| \item[job.services.{[id]}] Here {\em id} is one of the ids specified in the |
| job.services line above. The value is a weight, used to affect the distribution |
| of services among the jobs. |
| |
| \item[submission.spread] This is the time, in seconds the set of job submissions |
| is to be spread across. The jobs are submitted at random times such that the |
| total time between submitting the first job and the last job is approximately |
| this number. |
| \item[compression] For each sleep time in the job, divide the actual value by |
| this number. This allows testers to use the actual elapsed time from real |
| jobs, and compress the total run time so it fits approximately into the submission |
| spread. |
| |
| For example, if a collection of jobs was originally run over 24 hours, but |
| you want to run a simulation with approximately the same type of submission |
| that last only 15 minutes, specify a submission spread of 900 (15 minutes) and |
| a compression of 96. |
| \end{description} |
| |
| Here is a sample run configuration file: |
| \begin{verbatim} |
| # control file to create a random-like submission of jobs for batch submission |
| # This represents jobs submitted over approximately 36 hours real time |
| # Compression of 96 and spread 920 gives a good 15-20 minute test on test system with |
| # 136 15GB shares |
| |
| random.seed = 0 # a number, for determinate randoms |
| # or TOD, and the seed will use |
| # current time of day |
| |
| src.dir = jobs.in # where the jobs are |
| dest.dir = jobs # where to put prepared jobs |
| |
| scheduling.classes = normal # classes |
| scheduling.classes.normal = 100 |
| |
| job.memory = 28 37 # memorys to assign |
| job.memory.28 = 50 |
| job.memory.37 = 50 |
| |
| job.services = 0 1 2 3 4 5 6 7 |
| job.services.0 = 25 |
| job.services.1 = 25 |
| job.services.2 = 25 |
| job.services.3 = 25 |
| job.services.4 = 25 |
| job.services.5 = 25 |
| job.services.6 = 25 |
| job.services.7 = 25 |
| |
| submission.spread = 920 # number of *seconds* to try to spread submission over |
| |
| compression = 96 # comporession for timings |
| \end{verbatim} |
| |
| \subsection{Services} |
| \label{subsec:simulation.services} |
| It is possible to run the FixedSleepAE as a UIMA-AS service, with each job |
| specifying a dependency on the service, and the indicated service doing the |
| actual sleeping on behalf of the job. |
| |
| These variants on services are supported: |
| \begin{enumerate} |
| \item Registered services, started by reference, |
| \item Registered services, started by the simulator, |
| \end{enumerate} |
| |
| To use these simulated services, configure a ``service boot'' file and reference |
| the services from the job generation config file. |
| |
| Properties required in the service boot file include: |
| \begin{description} |
| \item[register] This specifies registered services. The value is a blank delimited |
| list of pseudo IDs for the registered services. |
| \item[start] This specifies which of the registered services to automatically |
| start. The value is some subset of the pseudo IDS specified under {\em register} |
| \item[instances\_{[id]}] Here {\em id} is one of the IDs specified for {\em submit, |
| register,} or {\em standalone}. The value is the number of instances of that |
| specific service to set up. |
| \end{description} |
| |
| \paragraph{Service pseudo IDs} |
| DUCC is packaged with 10 pre-configured services that use the FixedSleepAE. All of these |
| services behave identically, the only difference is their endpoints, which allows |
| the simulated runs to activate and use multiple independent services. Because the |
| endpoints are in the various UIMA XML service descriptors, it is necessary to use |
| exactly these IDs when generating a test run. Thus, the only valid pseudo-ids |
| for service configuration are {\em 0, 1, 2, 3, 4, 5, 6, 7, 8, 9}. |
| |
| These {\em service ids} are used on the job configuration file to establish a |
| weighted distribution of service use among the jobs. |
| |
| Here is a sample service configuration file: |
| \begin{verbatim} |
| # register these services, 2 instances each |
| register 0 1 2 3 |
| instances_0 2 |
| instances_1 2 |
| instances_2 2 |
| instances_3 2 |
| |
| # start these registered services |
| start 2 3 |
| \end{verbatim} |
| |
| \subsection{Generating a Job Set} |
| |
| The {\em prepare} script, found in \duccruntime/examples/systemtest is used |
| to generate a test run from the control files described above. |
| To use it, execute |
| \begin{verbatim} |
| prepare [config-file] |
| \end{verbatim} |
| where {\em config file} is the \hyperref[subsec:simulation.run-description]{run description} file |
| described above. |
| |
| This script reads the meta-specification in the {\em jobs.in} directive of the |
| config-file, generates a set of meta-specification files into the {\em jobs.out} |
| directory, and creates a control file, {\em job.ctl}. The {\em job.ctl} file is used |
| by the simulation driver to submit all the jobs. |
| |
| |
| \subsection{Running the Test Driver} |
| A test run is driven from the script {\em runducc} which resides in the |
| directory {\em \duccruntime/examples/systemtest}. This |
| script supports a large number of options intended to inject errors and otherwise |
| perturb a run. |
| |
| To use the test driver, first create a job collection as described above. This will |
| generate a file called {\em job.ctl} in the test directory containing the {\em prepare} |
| file. |
| |
| Then execute: |
| \begin{verbatim} |
| runducc -d jobdir -b batchfile options... |
| \end{verbatim} |
| where the various parameters and options include: |
| \begin{description} |
| \item[-d jobdir] The jobdir is the directory containing the {\em prepare} file and the |
| {\em job.ctl} file as describe in the previous section. |
| \item[-b batchfile] The batchfile is usually {\em job.ctl} as generated by the |
| prepare script. (This file may be hand-edited to create custom runs outside |
| of the {\em prepare} script.) |
| \item[--AE] This specifies to run all jobs as CR and AE. This is the default and |
| need not be specified. |
| \item[--DD] This specifies to run all jobs as CR and DD. The jobs are generated as |
| DD-style jobs, as opposed to AE. |
| \item[--SE cfg] This specifies to run all jobs using services, as generated by the {\em |
| prepare} script. The parameter is the \hyperref[subsec:simulation.services]{service |
| config file} as described above. When specified, the driver starts the services |
| as configured, pauses a bit to let them start up, and generated every job with a |
| dependency on one of the services. |
| \item[-i time-in-sec] If specified, this forces each AE to spend a minimum of the indicated time |
| in it's initialization method (also a sleep). If not specified, the default is |
| 10 seconds. The actual time is controlled by the {\em -r} (range) option. |
| \item[--init\_fail\_cap count] This sets the job property {\em process\_initialization\_failures\_cap} |
| to the the indicated value, to control the number of initialization failures to be tolerated |
| before terminating the job. |
| \item[--int\_timeout seconds] This sets the job property {\em process\_initialization\_time\_max} |
| to the indicated value, to control the time allowed in initialization before failure is reported. |
| \item[-r time-in-sec] This specifies the top range for initialization. The service |
| will spend the time specified in {\em -i}, PLUS a random value from 1 to |
| the time specified in {\em -r} in its initialization phase. |
| \item[--IB] The Job Process will leak memory in it's initialization phase until it is killed, hopefully by |
| DUCC, but possibly by the operating system. {\em Use with care.}. |
| \item[--PB] The job Process will leak memory in it's processing phase until it is killed, hopefully |
| by DUCC, but possibly by the operation system. {\em Use with care.} |
| \item[-m size-in-gb] Memory override. Use this value for all jobs, overriding the value |
| in the generated meta-specification file. |
| \item[-n max-Number-of-processes] Max machine override. If specified, this overrides the configured process max |
| from the job control file. Specify the max as $0$ and no maximum will be submitted with the job, |
| causing the scheduler to try to allocated the largest possible number of processes to the job. |
| \item[-p time-in-seconds] If specified the job property {\em process\_per\_item\_time\_max}, |
| which sets a timeout on work items, is set to the indicated time. |
| \item[-w, --watch] Submit every job with the {\em wait\_for\_completion} flag. This runs the |
| driver in multi-threaded mode, with each thread monitoring the progress of a job. |
| \item[-x rate] This specifies an expected error rate for execution phase in a job process, from 0-100 (a |
| percentage). When specified, each job process uses a random number generator to determine |
| the probability that is would crash, if if that probability is within the specified rate, it |
| generates a random exception. |
| \item[-y rate] This specifies an expected error rate for initialization phase in a job process, from 0-100 (a |
| percentage). When specified, each job process uses a random number generator to determine |
| the probability that is would crash, if if that probability is within the specified rate, it |
| generates a random exception. |
| \end{description} |
| |
| For an expected error-free run, only the -b and -d options are needed. |
| |
| |
| \section{Pre-Packaged Tests} |
| Three test suites are provided using the mechanisms described in the previous section: |
| \begin{itemize} |
| \item A 15-minute run comprising approximately 30 jobs. This includes configuration for |
| single-class submission, mixed class submission, and one configured to maximize |
| resource fragmentation. |
| \item A 30-minute run comprising approximately 33 jobs. This includes a single |
| configuration. |
| \item a 24-hour run comprising approximately 260 jobs. This also includes configurations |
| for single-class submission, mixed classes, and fragmentation. {\em Note: this run has |
| been reconfigured to run in 12 hours, and has been successfully been configured |
| to complete in 6 hours. This can create a significant load on the DUCC processes.} |
| \end{itemize} |
| |
| The configurations are found in the \duccruntime/examples/systemtest directory |
| and are in sub directories called, |
| \begin{itemize} |
| \item mega-15-min |
| \item mega-30-min |
| \item mega-24-hour |
| \end{itemize} |
| |
| To run these tests: |
| \begin{enumerate} |
| |
| \begin{sloppypar} |
| \item Create a node configuration. A sample configuration to generate |
| 52 simulated nodes, and which assumes the |
| physical machines for the simulation are called {\em sys290, sys291, sys292, sys293} |
| and {\em sys534} is supplied in \duccruntime/examples/systemtest. Change |
| the node names to the names of real machines, making any other adjustments |
| needed. |
| \end{sloppypar} |
| |
| \item Update your {\em \duccruntime/resources/ducc.nodes} so that all the real node names specified |
| int the simulated node file are included. |
| \item Update your {\em \duccruntime/resources/ducc.properties} so the |
| {\em ducc.head} is specified as the {\em real, physical} machine where you will |
| start the simulated cluster. |
| \item Be sure the {\em job driver} nodepool, if configured in |
| {\em \duccruntime/resources/ducc.classes}, specifies the name of one of the |
| simulated nodes. When first running these tests it is usually best that |
| the job driver NOT be configured on a specific node in {\em ducc.classes} |
| as it can be confusing to get this right on simulated clusters. |
| |
| Specifically, in {\em ducc.classes}, configure the {\em JobDriver} class |
| thus: |
| \begin{verbatim} |
| Class JobDriver fixed-base { } |
| \end{verbatim} |
| This allows DUCC to schedule the job driver on any node in the simulated |
| cluster. |
| |
| \item Generate the job set. For example, to generate the job set for the |
| 15-minute run, |
| \begin{verbatim} |
| cd $DUCC_HOME/examples/systemtest |
| ./prepare mega-15-min/jobs.prepare |
| \end{verbatim} |
| \item Start the simulated cluster (Assuming your simulated node file is called |
| {\em 52.simulated.nodes}: |
| \begin{verbatim} |
| cd $DUCC_HOME/examples/systemtest |
| ./start_sim -c all -n 52.simulated.nodes |
| \end{verbatim} |
| \item Use the webserver (or for advanced users, log files), to ensure |
| everything came up and the job driver node has been assigned. |
| \item Start the run: |
| \begin{verbatim} |
| cd $DUCC_HOME/examples/systemtest |
| ./runducc -d mega-15-min -b job.ctl |
| \end{verbatim} |
| \end{enumerate} |
| |
