| % |
| % Licensed to the Apache Software Foundation (ASF) under one |
| % or more contributor license agreements. See the NOTICE file |
| % distributed with this work for additional information |
| % regarding copyright ownership. The ASF licenses this file |
| % to you under the Apache License, Version 2.0 (the |
| % "License"); you may not use this file except in compliance |
| % with the License. You may obtain a copy of the License at |
| % |
| % http://www.apache.org/licenses/LICENSE-2.0 |
| % |
| % Unless required by applicable law or agreed to in writing, |
| % software distributed under the License is distributed on an |
| % "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| % KIND, either express or implied. See the License for the |
| % specific language governing permissions and limitations |
| % under the License. |
| % |
| % Create well-known link to this spot for HTML version |
| \ifpdf |
| \else |
| \HCode{<a name='DUCC_OVERVIEW'></a>} |
| \fi |
| \chapter{DUCC Overview} |
| |
| \section{What is DUCC?} |
| |
| DUCC stands for Distributed UIMA Cluster Computing. DUCC is a cluster management system |
| providing tooling, management, and scheduling facilities to automate the scale-out of |
| applications written to the UIMA framework. |
| |
| Core UIMA provides a generalized framework for applications that process unstructured |
| information such as human language, but does not provide a scale-out mechanism. UIMA-AS provides |
| a scale-out mechanism to distribute UIMA pipelines over a cluster of computing resources, but |
| does not provide job or cluster management of the resources. DUCC defines a formal job model |
| that closely maps to a standard UIMA pipeline. Around this job model DUCC provides cluster |
| management services to automate the scale-out of UIMA pipelines over computing clusters. |
| |
| \section{DUCC Job Model} |
| |
| The Job Model defines the steps necessary to scale-up a UIMA pipeline using DUCC. The goal of |
| DUCC is to allow the application logic to be unchanged. |
| |
| The DUCC Job model consists of standard UIMA components: a Collection Reader (CR), a CAS |
| Multiplier (CM), application logic as implemented one or more Analysis Engines (AE), and a CAS |
| Consumer (CC). In theory, any CR, or CM will work with DUCC, but DUCC is all about scale-out. In |
| order to achieve good scale-out these components must be constructed in a specific way. |
| |
| The Collection Reader builds input CASs and forwards them to the UIMA pipelines. In the DUCC |
| model, the CR is run in a process separate from the rest of the pipeline. In fact, in all but the |
| smallest clusters it is run on a different physical machine than the rest of the pipeline. To |
| achieve scalability, the CR must create very small CASs that do not contain application data, |
| but which contain references to data; for instance, file names. Ideally, the CR should be |
| runnable in a process not much larger than the smallest Java virtual machine. Later sections |
| demonstrate methods for achieving this. |
| |
| Each pipeline must contain at least one CAS Multiplier which receives the CASs from the CR. The |
| CMs encapsulate the knowledge of how to receive the data references in the small CASs received |
| from the CRs and deliver the referenced data to the application pipeline. DUCC packages the CM, |
| AE(s), and CC into a single process, multiple instances of which are then deployed over the |
| cluster. |
| |
| DUCC does not provide any mechanism for receiving output CASs. Each application must |
| supply its own CAS Consumer which serializes the output of the pipelines for |
| consumption by other entities (as serialized CASs, for example). |
| |
| A DUCC job therefore consists of a small specification containing the following items: |
| |
| \begin{itemize} |
| \item The name of a resource containing the CR descriptor. |
| \item The name of a resource containing the CM descriptor. |
| \item The name of a resource containing the AE descriptor. |
| \item The name of a resource containing the CC descriptor. |
| \item Other information required to parametrize the above and identify the job |
| such as log directory, working directory, desired scale-out, etc. These are |
| described in detail in subsequent sections. |
| \end{itemize} |
| |
| On job submission, DUCC examines the job specification and automatically creates a scaled-out |
| UIMA-AS service with a single process executing the CR as a UIMA-AS client and and as many |
| processes as possible executing the combined CM, AE, and CC pipeline as UIMA-AS service |
| instances. |
| |
| DUCC provides other facilities in support of scale-out: |
| \begin{itemize} |
| \item The ability to reserve all or part of a node in the cluster. |
| \item Automated management of services required in support of jobs. |
| \item The ability to schedule and execute arbitrary processes on nodes in the |
| cluster. |
| \item Debugging tools and support. |
| \item A web server to display and manage work and cluster status. |
| \item A CLI and a Java API to support the above. |
| \end{itemize} |
| |
| \section{DUCC From UIMA to Full Scale-out} |
| |
| In this section we demonstrate the progression of a simple UIMA pipeline to a fully |
| scaled-out job running under DUCC. |
| |
| \paragraph{UIMA Pipelines} |
| A normal UIMA pipeline |
| contains a Collection Reader, one or more Analysis Engines connected in a pipeline, and a CAS |
| Consumer as shown in \hyperref[fig:UIMA-pipeline]{Figure ~\ref{fig:UIMA-pipeline}}. |
| |
| \begin{figure}[H] |
| \centering |
| % \includegraphics[bb=0 0 575 310, width=5.5in]{images/uima-pipeline.jpg} |
| \includegraphics[width=5.5in]{images/uima-pipeline.jpg} |
| \caption{Standard UIMA Pipeline} |
| \label{fig:UIMA-pipeline} |
| \end{figure} |
| |
| \paragraph{UIMA-AS Scaled Pipeline} |
| With UIMA-AS the CR is separated into a discrete process and a CAS Multiplier is introduced |
| into the pipeline as an interface between the CR and the pipeline, as shown in |
| \hyperref[fig:UIMA-AS-pipeline]{Figure ~\ref{fig:UIMA-AS-pipeline}} below. |
| Multiple pipelines are serviced by the |
| CR and are scaled-out over a computing cluster. The difficulty with this model is that each |
| user is individually responsible for finding and scheduling computing nodes, installing |
| communication software such as ActiveMQ, and generally managing the distributed job and |
| associated hardware. |
| |
| \begin{figure}[H] |
| \centering |
| % \includegraphics[bb=0 0 584 341, width=5.5in]{images/uima-as-pipeline.jpg} |
| \includegraphics[width=5.5in]{images/uima-as-pipeline.jpg} |
| \caption{UIMA Pipeline As Scaled by UIMA-AS} |
| \label{fig:UIMA-AS-pipeline} |
| \end{figure} |
| |
| \paragraph{UIMA-AS Pipeline Scaled By DUCC} |
| DUCC is a UIMA and UIMA-AS-aware cluster manager. To scale out work under DUCC the developer |
| tells DUCC what the parts of the application are, and DUCC does the work to build the |
| scale-out via UIMA/AS, to find and schedule resources, to deploy the parts of the application |
| over the cluster, and to manage the jobs while it executes. |
| |
| On job submission, the DUCC Command Line Interface (CLI) inspects the XML defining the analytic |
| and generates a UIMA-AS Deployment Descriptor (DD). The DD establishes some number of pipeline |
| threads per process (as indicated in the DUCC job parameters), and generates job-unique queues. |
| |
| Under DUCC, the Collection Reader is executed in a process called the Job Driver (or JD). The |
| pipelines are executed in one or more processes called Job Processes (or JPs). The JD |
| process provides a thin wrapper over the CR to enable communication with DUCC. The JD uses the |
| CR to implement a UIMA-AS client delivering CASs to the multiple (scaled-out) pipelines, |
| shown in \hyperref[fig:UIMA-AS-pipeline-DUCC]{Figure ~\ref{fig:UIMA-AS-pipeline-DUCC}} below. |
| |
| \begin{figure}[H] |
| \centering |
| % \includegraphics[bb=0 0 571 311, width=5.5in]{images/ducc-sequential.jpg} |
| \includegraphics[width=5.5in]{images/ducc-sequential.jpg} |
| \caption{UIMA Pipeline As Automatically Scaled Out By DUCC} |
| \label{fig:UIMA-AS-pipeline-DUCC} |
| \end{figure} |
| |
| \paragraph{UIMA-AS Pipeline with User-Supplied DD Scaled By DUCC} |
| |
| Application programmers may supply their own Deployment Descriptors to control intra-process |
| threading and scale-out. If a DD is supplied in the job parameters, DUCC will use this instead |
| of generating one as depicted in \hyperref[fig:UIMA-AS-pipeline-DUCC-DD]{Figure ~\ref{fig:UIMA-AS-pipeline-DUCC-DD}} below. |
| |
| \begin{figure}[H] |
| \centering |
| % \includegraphics[bb=0 0 571 316,width=5.5in]{images/ducc-parallel.jpg} |
| \includegraphics[width=5.5in]{images/ducc-parallel.jpg} |
| \caption{UIMA Pipeline With User-Supplied DD as Automatically Scaled Out By DUCC} |
| \label{fig:UIMA-AS-pipeline-DUCC-DD} |
| \end{figure} |
| |
| |
| \section{Error Management } |
| DUCC provides a number of facilities to assist error management: |
| |
| \begin{itemize} |
| \item DUCC uses the UIMA-AS error-handling facilities to reflect errors from the Job Processes |
| to the Job Drivers. The JD wrappers implement logic to enforce error thresholds, to identify |
| and log errors, and to reflect job problems in the DUCC Web Server. All error thresholds are |
| configurable both globally and on a per-job basis. |
| |
| \item Error and timeout thresholds are implemented for both the initialization phase of a pipeline |
| and the execution phase. |
| |
| \item Retry-after-error is supported: if a process has a failure on some CAS after |
| initialization is successful, the process is terminated and all affected CASs are retried, up to some |
| configurable threshold. |
| |
| \item DUCC ensures that processes can successfully initialize before fully scaling out a job, |
| to ensure a cluster is not overwhelmed with errant processes. |
| |
| \item Various error conditions encountered while a job is running will prevent the errant job |
| from continuing scale out, and can result in termination of the job. |
| \end{itemize} |
| |
| \section{Cluster and Job Management} |
| DUCC supports management of multiple jobs and multiple users in a distributed cluster: |
| |
| \begin{description} |
| \item[Multiple User Support] DUCC runs all work under the identity of the submitting user. Logs |
| are written with the user's credentials into the user's file space designated at job |
| submission. |
| |
| \item[Fair-Share Scheduling] DUCC provides a Fair-Share scheduler to equitably share |
| resources among multiple users. The scheduler also supports semi-permanent reservation of |
| full or partial machines. |
| |
| \item[Service Management] DUCC provides a Service Manager capable of automatically starting, stopping, and |
| otherwise managing and querying both UIMA-AS and non-UIMA-AS services in support of jobs. |
| |
| \item[Job Lifetime Management and Orchestration] DUCC includes an Orchestrator to manage the |
| lifetimes of all entities in the system. |
| |
| \item[Node Sharing] DUCC allocates processes from one or more users on a node, each with a specified |
| amount of memory. DUCC's preferred mechanism for constraining memory use is Linux |
| Control Groups, or CGroups. For nodes that do not suport CGroups, DUCC agents monitor |
| RAM use and kill processes that exceed their share size by a settable fudge factor. |
| |
| \item[DUCC Agents] DUCC Agents manage each node's local resources and all |
| processes started by DUCC. Each node in a cluster has exactly one Agent. The Agent |
| \begin{itemize} |
| \item Monitors and reports node capabilities (memory, etc) and performance data (CPU busy, |
| swap, etc). |
| \item Starts, stops, and monitors all processes on behalf of users. |
| \item Patrols the node for ``foreign'' (non-DUCC) processes, reporting them to the |
| Web Server, and optionally reaping them. |
| \item Ensures job processes do not exceed their declared memory requirements |
| through the use of Linux Cgroups. |
| \end{itemize} |
| |
| \item[DUCC Web server] DUCC provides a web server displaying all aspects of the system: |
| \begin{itemize} |
| \item All jobs in the system, their current state, resource usage, etc. |
| |
| \item All reserved resources and associated information (owner, etc.), |
| including the ability to request and cancel reservations. |
| |
| \item All services, including the ability to start, stop, and modify |
| service definitions. |
| |
| \item All nodes in the system and their status, usage, etc. |
| |
| \item The status of all DUCC management processes. |
| |
| \item Access to documentation. |
| \end{itemize} |
| |
| |
| \item[Cluster Management Support] DUCC provides system management support to: |
| \begin{itemize} |
| \item Start, stop, and query full DUCC systems. |
| |
| \item Start, stop, and quiesce individual DUCC components. |
| |
| \item Add and delete nodes from the DUCC system. |
| |
| \item Discover DUCC processes (e.g. after partial failures). |
| |
| \item Find and kill errant job processes belonging to individual users. |
| |
| \item Monitor and display inter-DUCC messages. |
| \end{itemize} |
| \end{description} |
| |
| |
| \section{Security Measures} |
| The following DUCC security measures are provided: |
| |
| \begin{description} |
| \item[command line interface] The CLI employs HTTP to send requests |
| to the DUCC controller. The CLI creates and employs public and private |
| security keys in the user's home directory for authentication of HTTP |
| requests. The controller validates requests via these same security keys. |
| \item[webserver] The webserver facilitates operational control and |
| therefore authentication is desirable. |
| \begin{itemize} |
| \item[\textit{user}] Each user has the ability to control certain aspects of |
| only his/her active submissions. |
| \item[\textit{admin}] Each administrator has the ability to control certain |
| aspects of any user's active submissions, as well as modification of some |
| DUCC operational characteristics. |
| \end{itemize} |
| A simple interface is provided so |
| that an installation can plug-in a site specific authentication mechanism |
| comprising userid and password. |
| \item[ActiveMQ] TBD. |
| \end{description} |
| |
| \section{Security Issues} |
| The following DUCC security issues should be considered: |
| |
| \begin{description} |
| \item[submit transmission 'sniffed'] In the event that the DUCC submit |
| command is 'sniffed' then the user authentication mechanism is compromised |
| and user masquerading is possible. That is, the userid encryption mechanism |
| can be exploited such that user A can submit a job pretending to be user B. |
| \item[user \textit{ducc} password compromised] In the event that the \textit{ducc} |
| user password is compromised then the root privileged command |
| \textbf{ducc\_ling} can be used to become any other user except root. |
| \item[user \textit{root} password compromised] In the event that the |
| \textit{root} user password is compromised DUCC provides no protection. |
| That is, compromising the root user is equivalent to compromising the DUCC |
| user password. |
| \end{description} |
| |
