| % |
| % Licensed to the Apache Software Foundation (ASF) under one |
| % or more contributor license agreements. See the NOTICE file |
| % distributed with this work for additional information |
| % regarding copyright ownership. The ASF licenses this file |
| % to you under the Apache License, Version 2.0 (the |
| % "License"); you may not use this file except in compliance |
| % with the License. You may obtain a copy of the License at |
| % |
| % http://www.apache.org/licenses/LICENSE-2.0 |
| % |
| % Unless required by applicable law or agreed to in writing, |
| % software distributed under the License is distributed on an |
| % "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| % KIND, either express or implied. See the License for the |
| % specific language governing permissions and limitations |
| % under the License. |
| % |
| |
| \section{Administrative Commands} |
| |
| The administrative commands include a command to start DUCC, one to stop it, and one to |
| verify the configuration and query the state of the cluster. |
| |
| Note: The scripting that supports some of these functions runs (by default) in multi-threaded mode so |
| large clusters can be started, stopped, and queried quickly. If DUCC is running on an older |
| system, the threading may not work right, in which case the scripts detect this and run |
| single-threaded. As well, all these commands support a ``--nothreading'' option to manually |
| disable the threading. |
| |
| \subsection{start\_ducc} |
| \label{subsec:admin.start-ducc} |
| |
| \subsubsection{{\em Description}} |
| The command \ducchome/admin/start\_ducc is used to start DUCC processes. |
| It must be run from the head node. |
| If run with no parameters it takes the following actions: |
| \begin{itemize} |
| \item Starts the ActiveMQ server. |
| \item Starts the database. |
| \item Starts the management processes Resource Manager, Orchestrator, Process Manager, |
| Services Manager, and Web Server on the local node. |
| \item Starts an agent process on every node named in the default node list. |
| \end{itemize} |
| |
| \subsubsection{{\em Usage}} |
| |
| \begin{description} |
| \item[start\_ducc {[options]}] \hfill \\ |
| If no options are given, all DUCC processes are started, using the default node list, |
| {\em ducc.nodes}. |
| |
| \end{description} |
| |
| \subsubsection{{\em Options: }} |
| \begin{description} |
| |
| \item[-n, --nodelist {[nodefile] }] \hfill \\ |
| Start agents on the nodes in the nodefile. Multiple nodefiles may be specified: |
| \begin{verbatim} |
| start\_ducc -n foo.nodes -n bar.nodes -n baz.nodes |
| \end{verbatim} |
| |
| |
| \item[-c, --component {[component] }] \hfill \\ |
| Start a specific DUCC component, optionally on a specific node. If the component |
| name is qualified with a nodename, the component is started on that node. To qualify |
| a component name with a destination node, use the notation component@nodename. |
| Multiple components may be specified: |
| \begin{verbatim} |
| start\_ducc -c sm -c pm -c rm -c or@bj22 -c agent@n1 -c agent@n2 |
| \end{verbatim} |
| |
| Components include: |
| \begin{description} |
| \item[rm] The Resource Manager |
| \item[or]The Orchestrator |
| \item[pm]The Process Manager |
| \item[sm]The Service Manager |
| \item[ws]The Web Server |
| \item[agent@node]Node Agents |
| \item[broker] ActiveMQ broker |
| \item[db] Database |
| \end{description} |
| |
| \item[--nothreading] If specified, the command does not run in multi-threaded mode |
| even if it is supported on the local platform. |
| |
| \end{description} |
| |
| \subsubsection{{\em Notes: }} |
| A different nodelist may be used to specify where Agents are started. As well multiple node |
| lists may be specified, in which case Agents are started on all the nodes in the multiple node |
| lists. |
| |
| To start only agents, run start\_ducc specifying a nodelist explicitly. Note that the broker |
| must have already been started. |
| |
| To start a specific management process, run start\_ducc with the -c component parameter, |
| specify the component that should be started. |
| |
| \subsubsection{{\em Examples: }} |
| |
| Start some nodes from two different nodelist and start the broker. |
| \begin{verbatim} |
| start\_ducc -n foo.nodes -n bar.nodes -c broker |
| \end{verbatim} |
| |
| Start an agent on a specific node: |
| \begin{verbatim} |
| start\_ducc -c agent@a.specific.node |
| \end{verbatim} |
| |
| Start the webserver on node 'bingle': |
| \begin{verbatim} |
| start\_ducc -c ws@bingle |
| \end{verbatim} |
| |
| \subsubsection{{\em Debugging:}} |
| |
| Sometimes something will not start and it can be difficult to understand why. To diagnose, it is |
| helpful to know that {\em start\_ducc} is simply a wrapper around a lower-level bit of scripting |
| that does the actual work. That lower-level code can be invoked stand-alone, in which case |
| console messages that {\em start\_ducc} will have suppressed are presented to the console. |
| |
| The lower-level script is called {\em ducc.py} and accepts the same {\em -c component} flag as |
| start\_ducc. If some component will not start, try running {\em ducc.py -c component} directly. |
| It will start in the foreground and usually the cause of the problem becomes evident from |
| the console. |
| |
| For example, suppose the Resource Manager will not start. Run the following: |
| \begin{verbatim} |
| ./ducc.py -c rm |
| \end{verbatim} |
| and examine the output. Use {\em CTL-C} to stop the component when done. |
| |
| |
| \subsection{stop\_ducc} |
| \label{subsec:admin.stop-ducc} |
| |
| \subsubsection{{\em Description:}} |
| Stop\_ducc is used to stop DUCC processes. At least one parameter is required. |
| When {\em -a} is specified, the following actions are taken: |
| \begin{itemize} |
| \item Uses the ActiveMQ broker to broadcast a shutdown request to all |
| DUCC compoments, other than the ActiveMQ broker itself, and the database. |
| \item Waits a bit, for all daemons to stop. |
| \item Stops the database. |
| \item Stops the ActiveMQ broker. |
| \end{itemize} |
| |
| |
| \subsubsection{\em Usage:} |
| |
| \begin{description} |
| \item[stop\_ducc {[options]}] \hfill \\ |
| If no options are given, help text is presented. At least one option is required, to avoid |
| accidental cluster shutdown. |
| \end{description} |
| |
| |
| \subsubsection{Options:} |
| \begin{description} |
| |
| \item[-a --all] \hfill \\ |
| Stop all the DUCC processes, including agents and management processes. This |
| broadcasts a "shutdown" command to all DUCC processes. Shutdown is normally |
| performed gracefully with all processes given time to save state. |
| All user processes, both jobs and services, are sent shutdown signals. Job and service |
| processes which do not shutdown within a designated grace period are then forcibly |
| terminated with kill -9. |
| |
| \item[-n, --nodelist {[nodefile]}] \hfill \\ |
| Only the DUCC agents in the designated nodelists are shutdown. The processes are sent |
| kill -INT signals which triggers the Java shutdown hooks and enables graceful shutdown. |
| All user processes on the indicated nodes, both jobs and services, are sent "shutdown" |
| signals and are given a minute to shutdown gracefully. Job and service processes which do |
| not shutdown within a designated grace period are then forcibly terminated with kill -9. |
| |
| \begin{verbatim} |
| stop\_ducc -n foo.nodes -n bar.nodes -n baz.nodes |
| \end{verbatim} |
| |
| \item[-c, --component {[component]}] \hfill \\ |
| Stop a specific DUCC component. |
| |
| This may be used to stop an errant management component and subsequently restart it |
| (with start\_ducc). |
| |
| This may also be used to stop a specific agent and the job and services processes it is |
| managing, without the need to specify a nodelist. |
| |
| Examples: |
| |
| Stop agents on nodes n1 and n2: |
| \begin{verbatim} |
| stop_ducc -c agent@n1 -c agent@n2 |
| \end{verbatim} |
| |
| Stop and restart the rm: |
| \begin{verbatim} |
| stop_ducc -c rm |
| start_ducc -c rm |
| \end{verbatim} |
| |
| Components include: |
| \begin{description} |
| \item[rm] The Resource Manager. |
| \item[or] The Orchestrator. |
| \item[pm] The Process Manager. |
| \item[sm] The Service Manager. |
| \item[ws] The Web Server. |
| \item[db] The database. |
| \item[broker] The ActiveMQ broker (only if the broker is auto-managed). |
| \item[agent@node] Node Agent on the specified node. |
| \end{description} |
| |
| \item[-w, --wait {[time in seconds]}] If given, this signals the time to wait |
| after broadcasting the shutdown signal, and before stopping the ActiveMQ broker itself. |
| If not specified, the default is 60 seconds. |
| |
| NOTE: In production systems, it is generally wise to use the default of 60 seconds. For |
| test systems a shorter wait speeds cycle time. Be sure to use {\em check\_ducc -k} after |
| {\em stop\_ducc} if you change the wait time to insure all processes are actually stopped. |
| |
| \item[--nothreading] If specified, the command does not run in multi-threaded mode |
| even if it is supported on the local platform. |
| |
| \end{description} |
| |
| \subsubsection{{\em Notes:}} |
| Sometimes problems in the network or elsewhere prevent the DUCC components from stopping properly. The |
| {\em check\_ducc} command, described in the following section, contains options to query the |
| existance of DUCC processes in the cluster, to forcibly ({\em kill -9}) terminate them, and to |
| more gracefully terminate them ({\em kill -INT}). |
| |
| |
| |
| \subsection{check\_ducc} |
| \label{subsec:admin.check-ducc} |
| \subsubsection{{\em Description:}} |
| |
| Check\_ducc is used to verify the integrity of the DUCC installation and to find and report on |
| DUCC processes. It identifies processes owned by ducc (management processes, agents, |
| and job processes), and processes started by DUCC on behalf of users. |
| |
| Check\_ducc can also be used to clean up errant DUCC processes when stop\_ducc is unable |
| to do so. The difference is that stop\_ducc generally tries more gracefully stop processes. |
| check\_ducc is used as a last resort, or if a fast but graceless shutdown is desired. |
| |
| \subsubsection{\em{Usage: }} |
| |
| \begin{description} |
| \item[check\_ducc {[options]}] |
| If no options are given this is the equivalent of: |
| \begin{verbatim} |
| check_ducc -c -n ../resources/ducc.nodes |
| \end{verbatim} |
| |
| This verifies the integrity of the DUCC installation and searches for all the |
| processes owned by user {\em ducc} and started by DUCC on all the nodes in ducc.nodes. |
| \end{description} |
| |
| \subsubsection{\em{Options:}} |
| \begin{description} |
| \item[-n --nodelist {[nodefile]}] |
| Only the nodes specified in the nodefile are searched. The option may be specified |
| multiple times for multiple nodefiles. Note that the "local" node is always checked as well. |
| \begin{verbatim} |
| check_ducc -n nlist1 -n nlist2 |
| \end{verbatim} |
| |
| \item[-c --configuration] |
| Verify the \hyperref[sec:ducc.classes]{Resource Manager configuration}. |
| |
| \item[-p --pids] |
| Rewrite the PID file. The PID file contains the process ids of all known DUCC |
| management and agent processes. The PID file is normally managed by start\_ducc and |
| stop\_ducc and is stored in the file {\em ducc.pids} in directory {\em ducc\_runtime/state}. |
| |
| Occasionally the PID file can become partially or fully corrupted; for example, if a DUCC |
| process dies spontaneously. Use check\_ducc -p to search the cluster for processes and |
| refresh the PID file. |
| |
| \item[-i, --int] \hfill \\ |
| Use this to send a shutdown signal ({\em kill -INT}) to all the DUCC processes. The DUCC processes |
| catch this signal, close their resources and exit. Some resources take some time to close, or in |
| case of problems, are unable to close, in which case the DUCC processes will unconditionally exit. |
| |
| Sometimes problems in the network or elsewhere prevent {\em check\_ducc -i} from terminating |
| the DUCC processes. In this case, use {\em check\_ducc -k}, described below. |
| |
| \item[-k, --kill] \hfill \\ |
| Use this to forcibly kill a component using kill -9. This should only be used if {\em stop\_ducc} |
| or {\em check\_ducc -i} does not work. |
| |
| \item[--nothreading] If specified, the command does not run in multi-threaded mode |
| even if it is supported on the local platform. |
| |
| \item[-v, --verbose] \hfill \\ |
| When specified with {\em -c} to check the configuration, this emits a formatted version |
| of the node list showing the full structure of the scheduling classes. |
| |
| |
| \end{description} |
| |
| |
| \subsection{ducc\_post\_install} |
| \label{subsec:admin.ducc-post-install} |
| |
| \paragraph{Description:} |
| The post-installation script must be run only after the first installation of DUCC. |
| When updating an existing installation use \hyperref[subsec:admin.ducc-update]{\em ducc\_update}. |
| ducc\_post\_install performs these tasks: |
| \begin{enumerate} |
| \item Verifies that the correct level of Java and Python are installed and available. |
| \item Creates a default nodelist, \duccruntime/resources/ducc.nodes, containing the name of the node you are installing on. |
| \item Defines the ``ducc head'' node to be the node you are installing from. |
| \item Initializes the database. |
| \item Sets up the default https keystore for the webserver. |
| \item Installs the DUCC documentation ``ducc book'' into the DUCC webserver root. |
| \item Builds and installs the C program, ``ducc\_ling'', into the default location. |
| \item Ensures that the (supplied) ActiveMQ broker is runnable. |
| \end{enumerate} |
| |
| Once the script completes successfully \hyperref[subsec:admin.start-ducc]{\em start\_ducc} will run a single-user/unprivileged DUCC. |
| |
| \paragraph{Notes:} |
| If the script is rerun it will rename the previously created files so any customizations applied |
| can be recovered. |
| |
| \subsection{ducc\_update} |
| \label{subsec:admin.ducc-update} |
| |
| \paragraph{Description:} |
| This command is used to unpack a new release of DUCC and create a new installion or update |
| an existing one. |
| For a new installation it simply unpacks the tar file with the appropriate permissions. |
| The setup must be completed by running \hyperref[subsec:admin.ducc-post-install]{\em ducc\_post\_install}. |
| |
| When updating an existing installation it performs the following actions: |
| \begin{enumerate} |
| \item Checks that DUCC is not running. |
| \item Creates a site.ducc.properties file if updating from DUCC 1.1.0. |
| \item Creates a time-stamped archive directory to hold the old runtime. |
| \item Archives current files before updating them, except for the customizable ones. |
| \item Leaves in place any files added to the directories that may hold csite-specific files. |
| \item Reports which are replaced, added, or kept. |
| \item Rebuilds the non-privileged ducc\_ling. |
| \end{enumerate} |
| |
| The site-specific files, those holding customizations such as node and class definitions |
| as well as logs and job history, are left in place, |
| while all replaced files are archived under a folder called {\em ducc\_archives} |
| so the previous installation can be restored if necessary. |
| |
| Note that the update does not create the database. After updating to 2.1.0 from an earlier |
| version with the file-based persistent scheme the database should be created with |
| \hyperref[subsec:admin.db-create]{\em db\_create} |
| and the files holding state such as job history and service registrations loaded into the database with |
| \hyperref[subsec:admin.db-loader]{\em db\_loader}. |
| If this conversion is omitted DUCC will continue to use the file-based scheme but with some |
| loss of functionality that the database design would provide. |
| |
| \paragraph{Usage:} |
| This command takes two parameters, a pointer to the DUCC\_HOME to be updated or created, |
| and the name of the tar file containg the new build. |
| \begin{description} |
| \item[ducc\_update {\em some-ducc-home} {\em binary-tar-file}] |
| Update an existing DUCC installation or install a new one. |
| \end{description} |
| |
| \paragraph{Arguments:} |
| \begin{description} |
| \item[{\em some-ducc-home}] |
| This specifies the DUCC\_HOME you wish to create or update. If it doesn't exist a new |
| installation is created, otherwise it is updated. |
| \item[{\em binary-tar-file}] |
| The name of the binary tar file containing the new build. |
| \end{description} |
| |
| \paragraph{Example:} |
| \begin{verbatim} |
| ducc_update ducc_runtime /home/ducc/ducc_runtime Downloads/uima-ducc-2.1.0-bin.tar.gz |
| \end{verbatim} |
| |
| \subsection{rm\_reconfigure} |
| \label{subsec:admin.rm-reconfigure} |
| |
| \subsubsection{{\em Description:}} |
| Rm\_reconfigure is used to force the Resource Manager (RM) to reread all its configuration |
| files and reconfigure itself accordingly, without the need to fully stop and restart RM. |
| This is generally much faster than RM restart and avoids losing most state messages from |
| the other DUCC processes. |
| |
| The {\em rm\_reconfigure} command first performs a |
| \hyperref[sec:admin.properties-merge]{properties merge.} |
| |
| RM then validates the new |
| configuration, and if no errors are found, saves certain information such as current node |
| online-offline status. It then rereads the following configuration files and rebuilds its |
| internal structures accordingly: |
| \begin{itemize} |
| \item ducc.properties (after merging default.ducc.properties and site.ducc.properties, |
| \item ducc.classes, |
| \item log4j.xml. |
| \end{itemize} |
| The saved configuration is then restored into the newly configured structures. |
| On receipt of the next Orchestrator state, the RM fully rebuilds its state from the current |
| DUCC load and scheduling restarts. |
| |
| Depending on the nature of the new configuration, the current load may be adjusted; for |
| example, if the weight of a fair-share class is changed, preemptions or extra allocations |
| may be performed. |
| |
| If the new configuration is not consistent with the current load, a number of more drastic |
| adjustments will be performed: |
| \begin{itemize} |
| \item If a fair-share class is deleted, all existing jobs for that class are preempted |
| and a {\em refusal} message is sent to the Orchestrator for each affected job. |
| \item If a fair-share class is redefined over a different nodepool such that existing |
| work are no longer legally scheduled, any shares allocated over inappropriate |
| hosts are {\em preempted}. As soon as the preemptions are acknowledged, the RM |
| will reschedule the shares over the differently-configured resources. |
| \item If a non-preemptable class is deleted or reconfigured so existing non-preempt able |
| work is no longer allocated correctly, the following will occur: |
| \begin{itemize} |
| \item If the shares are for services, they are deallocated and a {\em refusal} is |
| sent to the Orchestrator. The Service Manager will observe this and restart the |
| processes, causing them to be reallocated over the changed configuration. |
| \item Otherwise, the RM leaves the allocation in place, but places the hosts on an |
| internal {\em blacklist}, preventing subsequent scheduling to those hosts. Once |
| the (now) incorrectly placed shares are freed (e.g. by canceling a reservation or |
| exit of a managed reservation), the hosts are again white listed and made available |
| for scheduling. |
| \end{itemize} |
| \end{itemize} |
| |
| In short, the RM makes every effort to avoid disturbing existing allocations, and blacklists |
| hosts that are no longer consistently configured for the current load, until such time as |
| the allocations on those hosts are released. |
| |
| \subsubsection{\em Usage:} |
| |
| \begin{description} |
| \item[rm\_reconfigure] \hfill \\ |
| This command has no options. |
| \end{description} |
| |
| |
| \iffalse % Dropped this script for 2.1 .... needs work |
| |
| \subsection{rm\_qload} |
| \label{subsec:admin.rm-qload} |
| |
| \subsubsection{{\em Description:}} |
| Rm\_qload is used to query the Resource Manager's scheduling tables to determine the |
| current demand and capacity of the system, as the RM sees it. The primary purpose |
| is to provide information to adjunct resource managers (such as a ``cloud'') to |
| determine the current needs, or lack thereof, of the system. The administrative |
| command is implemented as a Python script that interacts with the underlying |
| Java ``RmQueryLoadReply'' API and is provided mostly as an example of how |
| scripting can be used to interact with the RM. |
| |
| After displaying the current scheduling quantum, the response is provided in two sections: |
| \begin{enumerate} |
| \item Information showing the current demand and usage of resource classes, and |
| \item Information showing the current nodepool usage. |
| \end{enumerate} |
| |
| \subsubsection{\em Class section} |
| Three lines are emitted per class: |
| \begin{enumerate} |
| \item The name of the class and its scheduling policy, |
| \item A line showing the {\em demand}, or {\em request} by quantum, on the class, and |
| \item A line showing the {\em usage}, or {\em award}, by quantum on the class. |
| \end{enumerate} |
| |
| The numbers shown for {\em request} and {\em award} show the number of processes, by |
| memory, in terms of scheduling quantum, for each class. For example, assuming the |
| scheduling quantum is 15GB, the following shows: |
| \begin{itemize} |
| \item Five processes of quantum 2 (15-30GB) are requested, but only two have been awarded, |
| \item Three processes of quantum 3 (31-45GB) are requested and all have been awarded, |
| \item Four processes of quantum 4 (46-60GB) are requested, and two have been awarded. |
| \end{itemize} |
| \begin{verbatim} |
| Class normal policy FAIR_SHARE |
| requested 0 0 5 3 4 0 0 0 0 |
| awarded 0 0 2 3 2 0 0 0 0 |
| \end{verbatim} |
| |
| \subsubsection{\em Nodepool section} |
| Six lines are displayed for each nodepool: |
| \begin{enumerate} |
| \item The name of the nodepool, |
| \item A summary showing the number hosts in the pool which are online, dead (unresponsive), and |
| varied-off, the total quantum shares available to the nodepool, and the total unscheduled or |
| {\em free} shares. |
| \item The number of hosts known to the nodepool, by quantum, similar to the class listings above, |
| \item The nubmer of online hosts, by quantum, |
| \item The number of completely free hosts by quantum (no work currently scheduled), and |
| \item The number of {\em virtual} hosts, by quantum. A {\em virtual host} is created when a |
| host is partially scheduled. For example, if a 32G processes is scheduled on a 64G host, this |
| creates one free 32G {\em virtual host}. |
| \end{enumerate} |
| To determine the number of processes, by quantum, that can be scheduled, one must {\em sum} the |
| ``free'' and ``virtual'' columns. |
| |
| For example, (assuming a 15GB quantum), the following listing shows |
| that nodepool ``power'' contains fourteen hosts with at least 45GB each (3 quanta). Two |
| of these hosts have something scheduled on them (the ``free |
| machines'' line), leaving unused space of one 15G quantum on one |
| host, and one 30GB quantum on another host. |
| |
| \begin{verbatim} |
| Nodepool power |
| online 14 dead 0 offline 0 total-shares 42 free-shares 42 |
| all machines: 0 0 0 14 0 0 0 0 0 |
| online machines: 0 0 0 0 0 0 0 0 0 |
| free machines: 0 0 0 12 0 0 0 0 0 |
| virtual machines: 0 1 1 0 0 0 0 0 0 |
| \end{verbatim} |
| |
| \subsubsection{\em Usage:} |
| \begin{description} |
| \item[rm\_qload] \hfill \\ |
| This command has no options. |
| \end{description} |
| |
| \fi % End of dropped rm_qload |
| |
| \subsection{rm\_qoccupancy} |
| \label{subsec:admin.rm-qoccupancy} |
| |
| \subsubsection{{\em Description:}} |
| Rm\_qoccupancy provides a list of all known hosts to the RM, and for each host, the following information: |
| \begin{itemize} |
| \item The name of the host, |
| \item Whether the host has any blacklists processes on it, |
| \item Whether the host is currently onlline (responsive), |
| \item The status of the host; whether the host is schedulable ({\em up} or {\em down}. A responsive host becomes |
| unschedulable ({\em down}) if it is varied-off, |
| \item The nodepool the host is a member of, |
| \item The reported memory size of the host, |
| \item The {\em order} of the host. The {\em order} is defined to be the maximum number of quantum shares |
| supported by the host, |
| \item The number of unscheduled quantum shares on the host, and |
| \item If work is scheduled on the host, information relevent to that scheduled processes (or reservation). |
| \end{itemize} |
| |
| If work is scheduled on a host, the work summary is keyed thus: |
| \begin{description} |
| \item[J] The Orchestrator-assigned job id of the work, |
| \item[S] The RM-assigned share id of the work, |
| \item[O] The {\em order} of the allocation; that is, the number of quantum shares the allocation occupies, |
| \item[II] The {\em initialization investment}; the number of milliseconds the allocated work spent in its |
| initialization phase, if any (usually only UIMA-AS processes display this), |
| \item[IR] The {\em runtime investment}; the number of milliseconds spent processing the current CASs, if this |
| is a UIMA-AS processes. Note that this number can change dramatically, as it is the sum of time spent only |
| by the current CASs. When a CAS completes, it no longer contributes to the investment of the process. The RM |
| uses this information to determine the best candidate for eviction, if needed ot maintain fair-share. |
| \item[E] Whether the RM has preempted (evicted) the process but it has not yet exited, |
| \item[P] Whether the RM has purged the process (evicted, because the host is non-responsive), but it has not |
| been confirmed evicted, |
| \item[F] Whether the process is {\em fixed}; that is, non-preemptbable, |
| \item[I] Whether the initialization phase is completed (usually only UIMA-AS processes). |
| \end{description} |
| |
| The following example shows seven hosts, one with a preemptable share in the {\em --default--} |
| nodepool (on bluej290-5), and one with a non-preemptable share in the {\em jobdriver} nodepool. |
| \begin{verbatim} |
| Node Blacklisted Online Status Nodepool Memory Order Free |
| bluej290-5 False True up --default-- 32505856 2 0 |
| J[ 6006] S[ 189] O[2] II[ 0] IR[ 0] E[False] P[False] F[False] I[False] |
| |
| bluej290-6 False True up --default-- 32505856 2 2 |
| bluej290-7 False True up --default-- 32505856 2 2 |
| bluej291-26 False True up nightly-test 32505856 2 2 |
| bluej291-27 False True up nightly-test 32505856 2 2 |
| bluej293-60 False True up intel 32505856 2 2 |
| bluej537-73 False True up jobdriver 32505856 2 1 |
| J[ 5973] S[ 1] O[1] II[ 0] IR[ 0] E[False] P[False] F[ True] I[False] |
| |
| |
| \end{verbatim} |
| |
| |
| \subsubsection{\em Usage:} |
| |
| \begin{description} |
| \item[rm\_qoccupancy] \hfill \\ |
| This command has no options. |
| \end{description} |
| |
| |
| |
| \subsection{vary\_off} |
| \label{subsec:admin.vary-off} |
| \subsubsection{{\em Description:}} |
| |
| Vary\_off is used to remove a host from scheduling and to evict the preemptable work that is running on it. |
| This allows for graceful clearance of a host so the host can be take offline for maintenance, |
| or any other purpose (such as sharing the host with other applications.) |
| The DUCC agent is NOT stoppped; use \hyperref[subsec:admin.stop-ducc]{stop\_ducc} to stop the |
| agent. |
| Managed and unmanaged reservations are not canceled by {\em vary\_off}. |
| |
| Only the userid that started DUCC may issue {\em vary\_off}; attempts from other userids |
| are rejected. |
| |
| \subsubsection{\em{Usage: }} |
| |
| \begin{description} |
| \item[vary\_off list-of-hosts] |
| The {\em list-of-nodes} is a space delimited list of hosts to be removed from |
| scheduling in the DUCC cluster. |
| \end{description} |
| |
| \subsection{vary\_on} |
| \label{subsec:admin.vary-on} |
| \subsubsection{{\em Description:}} |
| |
| Vary\_on is used to restore a host to scheduling by DUCC. If the agent is still |
| alive the host becomes immediately available. The agent is not started by |
| {\em vary\_on}; use use |
| \hyperref[subsec:admin.start-ducc]{start\_ducc} to start the agent if needed. |
| |
| Only the userid that started DUCC may issue {\em vary\_on}; attempts from other userids |
| are rejected. |
| |
| \subsubsection{\em{Usage: }} |
| |
| \begin{description} |
| \item[vary\_on list-of-hosts] |
| The {\em list-of-nodes} is a space delimited list of hosts to be restored for |
| scheduling in the DUCC cluster. |
| \end{description} |
| |
| |
| \subsection{ducc\_properties\_manager} |
| \label{sec:cli.ducc-properties-manager} |
| |
| \paragraph{Description:} |
| This CLI is used to manually merge or difference two properties files. |
| |
| Normally, the DUCC scripts {\em start\_ducc, check\_ducc,}, and {\em rm\_configure} automatically |
| merge the file {\em default.ducc.properties} and {\em site.ducc.properties} when invoked. |
| |
| \paragraph{Usage:} |
| \begin{description} |
| \item[ducc\_props\_manager --merge file1 --with file2 --to file3] |
| Merge two properties files into one. Properties added to, or changed in, the second file |
| are used to override those in the first file, with the result written to the third file. |
| \item[ducc\_props\_manager --delta file1 --with file2 --to file3] |
| Compare two properties files and write the differences into a third file. The first file is |
| considered a ``master'' file. Properties with different values in the second file, or which |
| do not occur in the first file, are written into the third file. |
| \end{description} |
| |
| \paragraph{Options:} |
| \begin{description} |
| \item[$--$merge file1] |
| In this form, the two files specified in the {\em $--$with} and {\em$--$to} fields are merged, with the |
| results placed in $--$file3. Overrides are flagged with a change tag and the date of the merge. |
| |
| {\em file1} is considered the ``master'' properties file and is usually the unmodified file provided |
| with the DUCC distribution, {\em default.ducc.properties}. |
| |
| {\em file2} is considered a set of override or additional properties and is usually the site local |
| properties file, {\em site.ducc.properties.} |
| |
| \item[$--$delta file1] |
| In this form, the two files specified in the {\em $--$with} and {\em$--$to} fields are compared, with |
| differences placed in $--$file3. |
| |
| {\em file1} is considered the ``master'' properties file and is usually the unmodified file provided |
| with the DUCC distribution, {\em default.ducc.properties}. |
| |
| {\em file2} is considered the ``external'' properties file and is usually the properties file from |
| an older version of DUCC. |
| |
| Differences are placed in {\em $--$file3} which may be a viable first cut at a new {\em site.ducc.properties.} |
| |
| \item[$--$with file2] This specifies the properties file to merge with the master, or to difference |
| with the master properties file. |
| |
| \item[$--$to file3] This specifies the file to which the results of the merge or delta are written. |
| \end{description} |
| |
| \paragraph{Notes:} |
| None. |
| |
| \subsection{db\_create} |
| \label{subsec:admin.db-create} |
| |
| \paragraph{Description:} |
| This command is used to initialize the database. Normally the database is initialized |
| during {\em ducc\_post\_install} but if this is an existing DUCC installation that is |
| being migrated from a version that does not use the database, it will be necessary to |
| initialize the database with this command. |
| |
| This command performs the following actions: |
| \begin{enumerate} |
| \item Starts the database. |
| \item Disables the default database superuser. |
| \item Installs a database superuser as ``ducc'' and sets the password |
| to a random string. The password is saved |
| in DUCC\_HOME/resources.private/ducc.private.properties. |
| \item Installs the DUCC database schema. |
| \item Stops the database. |
| \end{enumerate} |
| |
| |
| This command takes no parameters. |
| |
| NOTE: The database user and password are NOT RELATED to any login ID on the system, |
| they are used and maintained by the database only. |
| |
| \subsection{db\_loader} |
| \label{subsec:admin.db-loader} |
| |
| \paragraph{Description:} |
| This command is used to copy the data from DUCC's older (pre 2.1.0) file-based persistence |
| into the database. The database schema must already exist, created either |
| with {\em ducc\_post\_install} or with {\em db\_create}. |
| |
| This command performs the following actions: |
| \begin{enumerate} |
| \item Starts the database. |
| \item Drops some of the indexes in the database. |
| \item Loads the Orchestrator checkpoint file from {\em DUCC\_HOME/state/orchestrator.chkpt}. |
| \item Loads all job history from {\em DUCC\_HOME/history/jobs}. |
| \item Loads all reservation history from {\em DUCC\_HOME/history/reservations}. |
| \item Loads all service instance and AP history from {\em DUCC\_HOME/history/services}. |
| \item Loads the service registry from {\em DUCC\_HOME/state/services}. |
| \item Loads the service registry histroy from {\em DUCC\_HOME/history/service-registry}. |
| \item Reloads the Orchestratory checkpoint, as a spot-check of the loader's instrumentation (to insure |
| load times stay reasonable.) |
| \item Re-installs the DUCC database schema. |
| \item Stops the database. |
| \item Optionally renames the file-based state so if you rerun the command, the data does not get reloaded. |
| \end{enumerate} |
| |
| When the command exits, DUCC should be ready to run with all its state in the database. |
| |
| This command takes two parameters, a pointer to the DUCC\_HOME you want to load from, and |
| a flag to disable the rename of the file-based state. |
| |
| \paragraph{Usage:} |
| \begin{description} |
| \item[db\_loader -i {\em some-ducc-home} {[--no-archive]}] |
| Load the database from the specified DUCC\_HOME, and optionally do not archive the original files |
| by renaming them. |
| \end{description} |
| |
| \paragraph{Options:} |
| \begin{description} |
| \item[$-i$ {\em some-ducc-home}] |
| This specifies the DUCC\_HOME you wish to load. Most of the time it is the DUCC\_HOME you |
| are running within, but it can be some other DUCC\_HOME if you have multiple installations and |
| want other history and state loaded. |
| \item[$--no-archive$] |
| If specified, the original files are not renamed. Note that only the directories in {\em history} |
| and {\em state} are renamed. To restore these, simply rename them back without the {\em archive} |
| suffix. |
| \end{description} |
| |
| \paragraph{Example:} |
| \begin{verbatim} |
| db_loader -i /home/ducc/ducc_runtime |
| db_loader -i /home/ducc.old/ducc_runtime --no-archive |
| \end{verbatim} |
| |
| \paragraph{Notes:} |
| The console shows progress of the loader. Full details of the load are written to a log {\em db-loader-log} |
| in the usual DUCC log directory, for reference and potential problem determination if something goes wrong. |
| |
