blob: 9f672606049b566c624ba989914e5d10190c371f [file] [log] [blame]
 % % Licensed to the Apache Software Foundation (ASF) under one % or more contributor license agreements. See the NOTICE file % distributed with this work for additional information % regarding copyright ownership. The ASF licenses this file % to you under the Apache License, Version 2.0 (the % "License"); you may not use this file except in compliance % with the License. You may obtain a copy of the License at % % http://www.apache.org/licenses/LICENSE-2.0 % % Unless required by applicable law or agreed to in writing, % software distributed under the License is distributed on an % "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY % KIND, either express or implied. See the License for the % specific language governing permissions and limitations % under the License. % \section{Administrative Commands} The administrative commands include a command to start DUCC, one to stop it, and one to verify the configuration and query the state of the cluster. Note: The scripting that supports some of these functions runs (by default) in multi-threaded mode so large clusters can be started, stopped, and queried quickly. If DUCC is running on an older system, the threading may not work right, in which case the scripts detect this and run single-threaded. As well, all these commands support a --nothreading'' option to manually disable the threading. \subsection{start\_ducc} \label{subsec:admin.start-ducc} \subsubsection{{\em Description}} The command \ducchome/admin/start\_ducc is used to start DUCC processes. It must be run from the head node. If run with no parameters it takes the following actions: \begin{itemize} \item Starts the ActiveMQ server. \item Starts the database. \item Starts the management processes Resource Manager, Orchestrator, Process Manager, Services Manager, and Web Server on the local node. \item Starts an agent process on every node named in the default node list. \end{itemize} \subsubsection{{\em Usage}} \begin{description} \item[start\_ducc {[options]}] \hfill \\ If no options are given, all DUCC processes are started, using the default node list, {\em ducc.nodes}. \end{description} \subsubsection{{\em Options: }} \begin{description} \item[-n, --nodelist {[nodefile] }] \hfill \\ Start agents on the nodes in the nodefile. Multiple nodefiles may be specified: \begin{verbatim} start\_ducc -n foo.nodes -n bar.nodes -n baz.nodes \end{verbatim} \item[-c, --component {[component] }] \hfill \\ Start a specific DUCC component, optionally on a specific node. If the component name is qualified with a nodename, the component is started on that node. To qualify a component name with a destination node, use the notation component@nodename. Multiple components may be specified: \begin{verbatim} start\_ducc -c sm -c pm -c rm -c or@bj22 -c agent@n1 -c agent@n2 \end{verbatim} Components include: \begin{description} \item[rm] The Resource Manager \item[or]The Orchestrator \item[pm]The Process Manager \item[sm]The Service Manager \item[ws]The Web Server \item[agent@node]Node Agents \item[broker] ActiveMQ broker \item[db] Database \end{description} \item[--nothreading] If specified, the command does not run in multi-threaded mode even if it is supported on the local platform. \end{description} \subsubsection{{\em Notes: }} A different nodelist may be used to specify where Agents are started. As well multiple node lists may be specified, in which case Agents are started on all the nodes in the multiple node lists. To start only agents, run start\_ducc specifying a nodelist explicitly. Note that the broker must have already been started. To start a specific management process, run start\_ducc with the -c component parameter, specify the component that should be started. \subsubsection{{\em Examples: }} Start some nodes from two different nodelist and start the broker. \begin{verbatim} start\_ducc -n foo.nodes -n bar.nodes -c broker \end{verbatim} Start an agent on a specific node: \begin{verbatim} start\_ducc -c agent@a.specific.node \end{verbatim} Start the webserver on node 'bingle': \begin{verbatim} start\_ducc -c ws@bingle \end{verbatim} \subsubsection{{\em Debugging:}} Sometimes something will not start and it can be difficult to understand why. To diagnose, it is helpful to know that {\em start\_ducc} is simply a wrapper around a lower-level bit of scripting that does the actual work. That lower-level code can be invoked stand-alone, in which case console messages that {\em start\_ducc} will have suppressed are presented to the console. The lower-level script is called {\em ducc.py} and accepts the same {\em -c component} flag as start\_ducc. If some component will not start, try running {\em ducc.py -c component} directly. It will start in the foreground and usually the cause of the problem becomes evident from the console. For example, suppose the Resource Manager will not start. Run the following: \begin{verbatim} ./ducc.py -c rm \end{verbatim} and examine the output. Use {\em CTL-C} to stop the component when done. \subsection{stop\_ducc} \label{subsec:admin.stop-ducc} \subsubsection{{\em Description:}} Stop\_ducc is used to stop DUCC processes. At least one parameter is required. When {\em -a} is specified, the following actions are taken: \begin{itemize} \item Uses the ActiveMQ broker to broadcast a shutdown request to all DUCC compoments, other than the ActiveMQ broker itself, and the database. \item Waits a bit, for all daemons to stop. \item Stops the database. \item Stops the ActiveMQ broker. \end{itemize} \subsubsection{\em Usage:} \begin{description} \item[stop\_ducc {[options]}] \hfill \\ If no options are given, help text is presented. At least one option is required, to avoid accidental cluster shutdown. \end{description} \subsubsection{Options:} \begin{description} \item[-a --all] \hfill \\ Stop all the DUCC processes, including agents and management processes. This broadcasts a "shutdown" command to all DUCC processes. Shutdown is normally performed gracefully with all processes given time to save state. All user processes, both jobs and services, are sent shutdown signals. Job and service processes which do not shutdown within a designated grace period are then forcibly terminated with kill -9. \item[-n, --nodelist {[nodefile]}] \hfill \\ Only the DUCC agents in the designated nodelists are shutdown. The processes are sent kill -INT signals which triggers the Java shutdown hooks and enables graceful shutdown. All user processes on the indicated nodes, both jobs and services, are sent "shutdown" signals and are given a minute to shutdown gracefully. Job and service processes which do not shutdown within a designated grace period are then forcibly terminated with kill -9. \begin{verbatim} stop\_ducc -n foo.nodes -n bar.nodes -n baz.nodes \end{verbatim} \item[-c, --component {[component]}] \hfill \\ Stop a specific DUCC component. This may be used to stop an errant management component and subsequently restart it (with start\_ducc). This may also be used to stop a specific agent and the job and services processes it is managing, without the need to specify a nodelist. Examples: Stop agents on nodes n1 and n2: \begin{verbatim} stop_ducc -c agent@n1 -c agent@n2 \end{verbatim} Stop and restart the rm: \begin{verbatim} stop_ducc -c rm start_ducc -c rm \end{verbatim} Components include: \begin{description} \item[rm] The Resource Manager. \item[or] The Orchestrator. \item[pm] The Process Manager. \item[sm] The Service Manager. \item[ws] The Web Server. \item[db] The database. \item[broker] The ActiveMQ broker (only if the broker is auto-managed). \item[agent@node] Node Agent on the specified node. \end{description} \item[-w, --wait {[time in seconds]}] If given, this signals the time to wait after broadcasting the shutdown signal, and before stopping the ActiveMQ broker itself. If not specified, the default is 60 seconds. NOTE: In production systems, it is generally wise to use the default of 60 seconds. For test systems a shorter wait speeds cycle time. Be sure to use {\em check\_ducc -k} after {\em stop\_ducc} if you change the wait time to insure all processes are actually stopped. \item[--nothreading] If specified, the command does not run in multi-threaded mode even if it is supported on the local platform. \end{description} \subsubsection{{\em Notes:}} Sometimes problems in the network or elsewhere prevent the DUCC components from stopping properly. The {\em check\_ducc} command, described in the following section, contains options to query the existance of DUCC processes in the cluster, to forcibly ({\em kill -9}) terminate them, and to more gracefully terminate them ({\em kill -INT}). \subsection{check\_ducc} \label{subsec:admin.check-ducc} \subsubsection{{\em Description:}} Check\_ducc is used to verify the integrity of the DUCC installation and to find and report on DUCC processes. It identifies processes owned by ducc (management processes, agents, and job processes), and processes started by DUCC on behalf of users. Check\_ducc can also be used to clean up errant DUCC processes when stop\_ducc is unable to do so. The difference is that stop\_ducc generally tries more gracefully stop processes. check\_ducc is used as a last resort, or if a fast but graceless shutdown is desired. \subsubsection{\em{Usage: }} \begin{description} \item[check\_ducc {[options]}] If no options are given this is the equivalent of: \begin{verbatim} check_ducc -c -n ../resources/ducc.nodes \end{verbatim} This verifies the integrity of the DUCC installation and searches for all the processes owned by user {\em ducc} and started by DUCC on all the nodes in ducc.nodes. \end{description} \subsubsection{\em{Options:}} \begin{description} \item[-n --nodelist {[nodefile]}] Only the nodes specified in the nodefile are searched. The option may be specified multiple times for multiple nodefiles. Note that the "local" node is always checked as well. \begin{verbatim} check_ducc -n nlist1 -n nlist2 \end{verbatim} \item[-c --configuration] Verify the \hyperref[sec:ducc.classes]{Resource Manager configuration}. \item[-p --pids] Rewrite the PID file. The PID file contains the process ids of all known DUCC management and agent processes. The PID file is normally managed by start\_ducc and stop\_ducc and is stored in the file {\em ducc.pids} in directory {\em ducc\_runtime/state}. Occasionally the PID file can become partially or fully corrupted; for example, if a DUCC process dies spontaneously. Use check\_ducc -p to search the cluster for processes and refresh the PID file. \item[-i, --int] \hfill \\ Use this to send a shutdown signal ({\em kill -INT}) to all the DUCC processes. The DUCC processes catch this signal, close their resources and exit. Some resources take some time to close, or in case of problems, are unable to close, in which case the DUCC processes will unconditionally exit. Sometimes problems in the network or elsewhere prevent {\em check\_ducc -i} from terminating the DUCC processes. In this case, use {\em check\_ducc -k}, described below. \item[-k, --kill] \hfill \\ Use this to forcibly kill a component using kill -9. This should only be used if {\em stop\_ducc} or {\em check\_ducc -i} does not work. \item[--nothreading] If specified, the command does not run in multi-threaded mode even if it is supported on the local platform. \item[-v, --verbose] \hfill \\ When specified with {\em -c} to check the configuration, this emits a formatted version of the node list showing the full structure of the scheduling classes. \end{description} \subsection{ducc\_post\_install} \label{subsec:admin.ducc-post-install} \paragraph{Description:} The post-installation script must be run only after the first installation of DUCC. When updating an existing installation use \hyperref[subsec:admin.ducc-update]{\em ducc\_update}. ducc\_post\_install performs these tasks: \begin{enumerate} \item Verifies that the correct level of Java and Python are installed and available. \item Creates a default nodelist, \duccruntime/resources/ducc.nodes, containing the name of the node you are installing on. \item Defines the ducc head'' node to be the node you are installing from. \item Initializes the database. \item Sets up the default https keystore for the webserver. \item Installs the DUCC documentation ducc book'' into the DUCC webserver root. \item Builds and installs the C program, ducc\_ling'', into the default location. \item Ensures that the (supplied) ActiveMQ broker is runnable. \end{enumerate} Once the script completes successfully \hyperref[subsec:admin.start-ducc]{\em start\_ducc} will run a single-user/unprivileged DUCC. \paragraph{Notes:} If the script is rerun it will rename the previously created files so any customizations applied can be recovered. \subsection{ducc\_update} \label{subsec:admin.ducc-update} \paragraph{Description:} This command is used to unpack a new release of DUCC and create a new installion or update an existing one. For a new installation it simply unpacks the tar file with the appropriate permissions. The setup must be completed by running \hyperref[subsec:admin.ducc-post-install]{\em ducc\_post\_install}. When updating an existing installation it performs the following actions: \begin{enumerate} \item Checks that DUCC is not running. \item Creates a site.ducc.properties file if updating from DUCC 1.1.0. \item Creates a time-stamped archive directory to hold the old runtime. \item Archives current files before updating them, except for the customizable ones. \item Leaves in place any files added to the directories that may hold csite-specific files. \item Reports which are replaced, added, or kept. \item Rebuilds the non-privileged ducc\_ling. \end{enumerate} The site-specific customized files (i.e. those holding node and class definitions, passwords etc.) are left in place, while all replaced files are archived under a folder called {\em ducc\_archives} so the previous installation can be restored if necessary. Note that the update does not create the database. After updating to 2.1.0 from an earlier version with the file-based persistent scheme the database should be created with \hyperref[subsec:admin.db-create]{\em db\_create} and the files holding state such as job history and service registrations loaded into the database with \hyperref[subsec:admin.db-loader]{\em db\_loader}. If this conversion is omitted DUCC will continue to use the file-based scheme but with some loss of functionality that the database design would provide. \paragraph{Usage:} This command takes two parameters, a pointer to the DUCC\_HOME to be updated or created, and the name of the tar file containg the new build. \begin{description} \item[ducc\_update {\em some-ducc-home} {\em binary-tar-file}] Update an existing DUCC installation or install a new one. \end{description} \paragraph{Arguments:} \begin{description} \item[{\em some-ducc-home}] This specifies the DUCC\_HOME you wish to create or update. If it doesn't exist a new installation is created, otherwise it is updated. \item[{\em binary-tar-file}] The name of the binary tar file containing the new build. \end{description} \paragraph{Example:} \begin{verbatim} ducc_update ducc_runtime /home/ducc/ducc_runtime Downloads/uima-ducc-2.1.0-bin.tar.gz \end{verbatim} \subsection{rm\_reconfigure} \label{subsec:admin.rm-reconfigure} \subsubsection{{\em Description:}} Rm\_reconfigure is used to force the Resource Manager (RM) to reread all its configuration files and reconfigure itself accordingly, without the need to fully stop and restart RM. This is generally much faster than RM restart and avoids losing most state messages from the other DUCC processes. The {\em rm\_reconfigure} command first performs a \hyperref[sec:admin.properties-merge]{properties merge.} RM then validates the new configuration, and if no errors are found, saves certain information such as current node online-offline status. It then rereads the following configuration files and rebuilds its internal structures accordingly: \begin{itemize} \item ducc.properties (after merging default.ducc.properties and site.ducc.properties, \item ducc.classes, \item log4j.xml. \end{itemize} The saved configuration is then restored into the newly configured structures. On receipt of the next Orchestrator state, the RM fully rebuilds its state from the current DUCC load and scheduling restarts. Depending on the nature of the new configuration, the current load may be adjusted; for example, if the weight of a fair-share class is changed, preemptions or extra allocations may be performed. If the new configuration is not consistent with the current load, a number of more drastic adjustments will be performed: \begin{itemize} \item If a fair-share class is deleted, all existing jobs for that class are preempted and a {\em refusal} message is sent to the Orchestrator for each affected job. \item If a fair-share class is redefined over a different nodepool such that existing work are no longer legally scheduled, any shares allocated over inappropriate hosts are {\em preempted}. As soon as the preemptions are acknowledged, the RM will reschedule the shares over the differently-configured resources. \item If a non-preemptable class is deleted or reconfigured so existing non-preempt able work is no longer allocated correctly, the following will occur: \begin{itemize} \item If the shares are for services, they are deallocated and a {\em refusal} is sent to the Orchestrator. The Service Manager will observe this and restart the processes, causing them to be reallocated over the changed configuration. \item Otherwise, the RM leaves the allocation in place, but places the hosts on an internal {\em blacklist}, preventing subsequent scheduling to those hosts. Once the (now) incorrectly placed shares are freed (e.g. by canceling a reservation or exit of a managed reservation), the hosts are again white listed and made available for scheduling. \end{itemize} \end{itemize} In short, the RM makes every effort to avoid disturbing existing allocations, and blacklists hosts that are no longer consistently configured for the current load, until such time as the allocations on those hosts are released. \subsubsection{\em Usage:} \begin{description} \item[rm\_reconfigure] \hfill \\ This command has no options. \end{description} \iffalse % Dropped this script for 2.1 .... needs work \subsection{rm\_qload} \label{subsec:admin.rm-qload} \subsubsection{{\em Description:}} Rm\_qload is used to query the Resource Manager's scheduling tables to determine the current demand and capacity of the system, as the RM sees it. The primary purpose is to provide information to adjunct resource managers (such as a cloud'') to determine the current needs, or lack thereof, of the system. The administrative command is implemented as a Python script that interacts with the underlying Java RmQueryLoadReply'' API and is provided mostly as an example of how scripting can be used to interact with the RM. After displaying the current scheduling quantum, the response is provided in two sections: \begin{enumerate} \item Information showing the current demand and usage of resource classes, and \item Information showing the current nodepool usage. \end{enumerate} \subsubsection{\em Class section} Three lines are emitted per class: \begin{enumerate} \item The name of the class and its scheduling policy, \item A line showing the {\em demand}, or {\em request} by quantum, on the class, and \item A line showing the {\em usage}, or {\em award}, by quantum on the class. \end{enumerate} The numbers shown for {\em request} and {\em award} show the number of processes, by memory, in terms of scheduling quantum, for each class. For example, assuming the scheduling quantum is 15GB, the following shows: \begin{itemize} \item Five processes of quantum 2 (15-30GB) are requested, but only two have been awarded, \item Three processes of quantum 3 (31-45GB) are requested and all have been awarded, \item Four processes of quantum 4 (46-60GB) are requested, and two have been awarded. \end{itemize} \begin{verbatim} Class normal policy FAIR_SHARE requested 0 0 5 3 4 0 0 0 0 awarded 0 0 2 3 2 0 0 0 0 \end{verbatim} \subsubsection{\em Nodepool section} Six lines are displayed for each nodepool: \begin{enumerate} \item The name of the nodepool, \item A summary showing the number hosts in the pool which are online, dead (unresponsive), and varied-off, the total quantum shares available to the nodepool, and the total unscheduled or {\em free} shares. \item The number of hosts known to the nodepool, by quantum, similar to the class listings above, \item The nubmer of online hosts, by quantum, \item The number of completely free hosts by quantum (no work currently scheduled), and \item The number of {\em virtual} hosts, by quantum. A {\em virtual host} is created when a host is partially scheduled. For example, if a 32G processes is scheduled on a 64G host, this creates one free 32G {\em virtual host}. \end{enumerate} To determine the number of processes, by quantum, that can be scheduled, one must {\em sum} the free'' and virtual'' columns. For example, (assuming a 15GB quantum), the following listing shows that nodepool power'' contains fourteen hosts with at least 45GB each (3 quanta). Two of these hosts have something scheduled on them (the free machines'' line), leaving unused space of one 15G quantum on one host, and one 30GB quantum on another host. \begin{verbatim} Nodepool power online 14 dead 0 offline 0 total-shares 42 free-shares 42 all machines: 0 0 0 14 0 0 0 0 0 online machines: 0 0 0 0 0 0 0 0 0 free machines: 0 0 0 12 0 0 0 0 0 virtual machines: 0 1 1 0 0 0 0 0 0 \end{verbatim} \subsubsection{\em Usage:} \begin{description} \item[rm\_qload] \hfill \\ This command has no options. \end{description} \fi % End of dropped rm_qload \subsection{rm\_qoccupancy} \label{subsec:admin.rm-qoccupancy} \subsubsection{{\em Description:}} Rm\_qoccupancy provides a list of all known hosts to the RM, and for each host, the following information: \begin{itemize} \item The name of the host, \item Whether the host has any blacklists processes on it, \item Whether the host is currently onlline (responsive), \item The status of the host; whether the host is schedulable ({\em up} or {\em down}. A responsive host becomes unschedulable ({\em down}) if it is varied-off, \item The nodepool the host is a member of, \item The reported memory size of the host, \item The {\em order} of the host. The {\em order} is defined to be the maximum number of quantum shares supported by the host, \item The number of unscheduled quantum shares on the host, and \item If work is scheduled on the host, information relevent to that scheduled processes (or reservation). \end{itemize} If work is scheduled on a host, the work summary is keyed thus: \begin{description} \item[J] The Orchestrator-assigned job id of the work, \item[S] The RM-assigned share id of the work, \item[O] The {\em order} of the allocation; that is, the number of quantum shares the allocation occupies, \item[II] The {\em initialization investment}; the number of milliseconds the allocated work spent in its initialization phase, if any (usually only UIMA-AS processes display this), \item[IR] The {\em runtime investment}; the number of milliseconds spent processing the current CASs, if this is a UIMA-AS processes. Note that this number can change dramatically, as it is the sum of time spent only by the current CASs. When a CAS completes, it no longer contributes to the investment of the process. The RM uses this information to determine the best candidate for eviction, if needed ot maintain fair-share. \item[E] Whether the RM has preempted (evicted) the process but it has not yet exited, \item[P] Whether the RM has purged the process (evicted, because the host is non-responsive), but it has not been confirmed evicted, \item[F] Whether the process is {\em fixed}; that is, non-preemptbable, \item[I] Whether the initialization phase is completed (usually only UIMA-AS processes). \end{description} The following example shows seven hosts, one with a preemptable share in the {\em --default--} nodepool (on bluej290-5), and one with a non-preemptable share in the {\em jobdriver} nodepool. \begin{verbatim} Node Blacklisted Online Status Nodepool Memory Order Free bluej290-5 False True up --default-- 32505856 2 0 J[ 6006] S[ 189] O[2] II[ 0] IR[ 0] E[False] P[False] F[False] I[False] bluej290-6 False True up --default-- 32505856 2 2 bluej290-7 False True up --default-- 32505856 2 2 bluej291-26 False True up nightly-test 32505856 2 2 bluej291-27 False True up nightly-test 32505856 2 2 bluej293-60 False True up intel 32505856 2 2 bluej537-73 False True up jobdriver 32505856 2 1 J[ 5973] S[ 1] O[1] II[ 0] IR[ 0] E[False] P[False] F[ True] I[False] \end{verbatim} \subsubsection{\em Usage:} \begin{description} \item[rm\_qoccupancy] \hfill \\ This command has no options. \end{description} \subsection{vary\_off} \label{subsec:admin.vary-off} \subsubsection{{\em Description:}} Vary\_off is used to remove a host from scheduling and to evict the preemptable work that is running on it. This allows for graceful clearance of a host so the host can be take offline for maintenance, or any other purpose (such as sharing the host with other applications.) The DUCC agent is NOT stoppped; use \hyperref[subsec:admin.stop-ducc]{stop\_ducc} to stop the agent. Managed and unmanaged reservations are not canceled by {\em vary\_off}. Only the userid that started DUCC may issue {\em vary\_off}; attempts from other userids are rejected. \subsubsection{\em{Usage: }} \begin{description} \item[vary\_off list-of-hosts] The {\em list-of-nodes} is a space delimited list of hosts to be removed from scheduling in the DUCC cluster. \end{description} \subsection{vary\_on} \label{subsec:admin.vary-on} \subsubsection{{\em Description:}} Vary\_on is used to restore a host to scheduling by DUCC. If the agent is still alive the host becomes immediately available. The agent is not started by {\em vary\_on}; use use \hyperref[subsec:admin.start-ducc]{start\_ducc} to start the agent if needed. Only the userid that started DUCC may issue {\em vary\_on}; attempts from other userids are rejected. \subsubsection{\em{Usage: }} \begin{description} \item[vary\_on list-of-hosts] The {\em list-of-nodes} is a space delimited list of hosts to be restored for scheduling in the DUCC cluster. \end{description} \subsection{ducc\_properties\_manager} \label{sec:cli.ducc-properties-manager} \paragraph{Description:} This CLI is used to manually merge or difference two properties files. Normally, the DUCC scripts {\em start\_ducc, check\_ducc,}, and {\em rm\_configure} automatically merge the file {\em default.ducc.properties} and {\em site.ducc.properties} when invoked. \paragraph{Usage:} \begin{description} \item[ducc\_props\_manager --merge file1 --with file2 --to file3] Merge two properties files into one. Properties added to, or changed in, the second file are used to override those in the first file, with the result written to the third file. \item[ducc\_props\_manager --delta file1 --with file2 --to file3] Compare two properties files and write the differences into a third file. The first file is considered a master'' file. Properties with different values in the second file, or which do not occur in the first file, are written into the third file. \end{description} \paragraph{Options:} \begin{description} \item[$--$merge file1] In this form, the two files specified in the {\em $--$with} and {\em$--$to} fields are merged, with the results placed in $--$file3. Overrides are flagged with a change tag and the date of the merge. {\em file1} is considered the master'' properties file and is usually the unmodified file provided with the DUCC distribution, {\em default.ducc.properties}. {\em file2} is considered a set of override or additional properties and is usually the site local properties file, {\em site.ducc.properties.} \item[$--$delta file1] In this form, the two files specified in the {\em $--$with} and {\em$--$to} fields are compared, with differences placed in $--$file3. {\em file1} is considered the master'' properties file and is usually the unmodified file provided with the DUCC distribution, {\em default.ducc.properties}. {\em file2} is considered the external'' properties file and is usually the properties file from an older version of DUCC. Differences are placed in {\em $--$file3} which may be a viable first cut at a new {\em site.ducc.properties.} \item[$--$with file2] This specifies the properties file to merge with the master, or to difference with the master properties file. \item[$--$to file3] This specifies the file to which the results of the merge or delta are written. \end{description} \paragraph{Notes:} None. \subsection{db\_create} \label{subsec:admin.db-create} \paragraph{Description:} This command is used to initialize the database. Normally the database is initialized during {\em ducc\_post\_install} but if this is an existing DUCC installation that is being migrated from a version that does not use the database, it will be necessary to initialize the database with this command. This command performs the following actions: \begin{enumerate} \item Starts the database. \item Disables the default database superuser. \item Installs a database superuser as ducc'' and sets the password to a random string. The password is saved in DUCC\_HOME/resources.private/ducc.private.properties. \item Installs the DUCC database schema. \item Stops the database. \end{enumerate} This command takes no parameters. NOTE: The database user and password are NOT RELATED to any login ID on the system, they are used and maintained by the database only. \subsection{db\_loader} \label{subsec:admin.db-loader} \paragraph{Description:} This command is used to copy the data from DUCC's older (pre 2.1.0) file-based persistence into the database. The database schema must already exist, created either with {\em ducc\_post\_install} or with {\em db\_create}. This command performs the following actions: \begin{enumerate} \item Starts the database. \item Drops some of the indexes in the database. \item Loads the Orchestrator checkpoint file from {\em DUCC\_HOME/state/orchestrator.chkpt}. \item Loads all job history from {\em DUCC\_HOME/history/jobs}. \item Loads all reservation history from {\em DUCC\_HOME/history/reservations}. \item Loads all service instance and AP history from {\em DUCC\_HOME/history/services}. \item Loads the service registry from {\em DUCC\_HOME/state/services}. \item Loads the service registry histroy from {\em DUCC\_HOME/history/service-registry}. \item Reloads the Orchestratory checkpoint, as a spot-check of the loader's instrumentation (to insure load times stay reasonable.) \item Re-installs the DUCC database schema. \item Stops the database. \item Optionally renames the file-based state so if you rerun the command, the data does not get reloaded. \end{enumerate} When the command exits, DUCC should be ready to run with all its state in the database. This command takes two parameters, a pointer to the DUCC\_HOME you want to load from, and a flag to disable the rename of the file-based state. \paragraph{Usage:} \begin{description} \item[db\_loader -i {\em some-ducc-home} {[--no-archive]}] Load the database from the specified DUCC\_HOME, and optionally do not archive the original files by renaming them. \end{description} \paragraph{Options:} \begin{description} \item[$-i$ {\em some-ducc-home}] This specifies the DUCC\_HOME you wish to load. Most of the time it is the DUCC\_HOME you are running within, but it can be some other DUCC\_HOME if you have multiple installations and want other history and state loaded. \item[$--no-archive$] If specified, the original files are not renamed. Note that only the directories in {\em history} and {\em state} are renamed. To restore these, simply rename them back without the {\em archive} suffix. \end{description} \paragraph{Example:} \begin{verbatim} db_loader -i /home/ducc/ducc_runtime db_loader -i /home/ducc.old/ducc_runtime --no-archive \end{verbatim} \paragraph{Notes:} The console shows progress of the loader. Full details of the load are written to a log {\em db-loader-log} in the usual DUCC log directory, for reference and potential problem determination if something goes wrong.