% 
% Licensed to the Apache Software Foundation (ASF) under one
% or more contributor license agreements.  See the NOTICE file
% distributed with this work for additional information
% regarding copyright ownership.  The ASF licenses this file
% to you under the Apache License, Version 2.0 (the
% "License"); you may not use this file except in compliance
% with the License.  You may obtain a copy of the License at
% 
%   http://www.apache.org/licenses/LICENSE-2.0
% 
% Unless required by applicable law or agreed to in writing,
% software distributed under the License is distributed on an
% "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
% KIND, either express or implied.  See the License for the
% specific language governing permissions and limitations
% under the License.
% 

\input{part5/ducc-pops-defs.tex}

% body

\chapter{Platform}

    The \varDistributedUIMAClusterComputing~(\varDUCC) platform comprises software 
    designed to facilitate the scale-out of 
    \varUnstructuredInformationManagementArchitecture~(\varUIMA) pipelines on a 
    collection of \varNodesMachinesComputers~shared "fairly" by a group of users.
    
    The major components of \varDUCC~are the \varOrchestrator~(\varOR), the \varJobDriver~(\varJD), 
    the \varResourceManager~(\varRM), the \varProcessManager~(\varPM), the \varServicesManager~(\varSM), 
    the \varAgents, the \varCommandLineInterface~(\varCLI), the \varApplicationProgramInterface~(\varAPI), 
    and the \varWebServer~(\varWS).
        
    \section{Highlights}
    
    \varDUCC~was conceived to address the following:
    
    \begin{itemize}
      \item manage a cluster of machines for \varUIMA~workloads
      \item highly configurable "fair-share" resource allocation system
      \item application code runs with credentials of submitting user
      \item "virtual machine" resources for user processes allocated instantaneously via \varLinuxControlGroups
      \item extensive Web, \varCLI~and \varAPI~interfaces
      \item rich debugging support for user processes
    \end{itemize} 
    
    \section{Architecture}
    
    The \varDUCC~platform employs building-block software from the open-source community
    where possible to achieve it goals. Foremost and not surprisingly \varDUCC~employs in
    its foundation \varUIMAAS, which in-turn relies upon \varUIMA-Core.
    
    Additionally, \varCamel~is used for inter-component communications.
    \varActiveMQ~is employed to process work items amongst a distributed set of work item processors.
    Logging is facilitated by \varLogger. \varJetty~is used for the \varWebServer~and \varjQuery~is deployed
    to web browsers.  And various other open-source softwares are likewise employed.
    
    By employing reliable open-source code where possible, the amount of custom code needed
    to develop and maintain \varDUCC~functionality is minimized. And substitution of implementation
    for equivalent functionality is possible, for example replacing \varApacheActiveMQ~with 
    \varIBMWebSphereMQ.
    
    \begin{figure}[H]
      \centering
      \includegraphics[width=6.5in]{images/ducc-arch.jpg}
      \label{fig:DUCC-Architecture}
    \end{figure}
    
%    \begin{figure}[h]
%    \centering
%    \includegraphics[width=6.5in]{images/ducc-arch.jpg}
%    \caption[]{\varDUCC~Architecture}
%    \label{fig:\varDUCC~Architecture}
%    \end{figure}
%    \label{fig:01}

    \section{\varJobs}

    The main focus of the system is running "batch" \varJobs~comprising \varUIMA~pipelines.
    
    Users submit \varJobs~to the system to be deployed and executed. \varJobs~have a
    life-cycle from birth to death during which time a (normally) finite collection of
    work items are processed by one or more \varUIMA~pipelines. \varJobs~consist of two
    parts: a singleton work item supplier, known in \varUIMA~parlance as a
    \varCollectionReader~(\varCR); and one or more pipelines, each known in \varUIMA~parlance
    as an \varAnalysisEngine~(\varAE).
        
    \subsection{Characteristics}   
    
    \varDUCC~facilitates "fair-share" \varUIMA~pipeline scale-out.
    
    The \varUIMA~pipelines comprising a \varJob~represent "embarrassingly parallel" 
    deployments. Over time, a \varJob~may expand and contract with respect to the number of 
    \varUIMA~pipelines deployed during its lifetime. This may be due to the introduction 
    or completion of other \varJobs, the rise and fall of other resource consumers such 
    as \varReservations~or \varServices, and the addition or removal of computer resources 
    to the cluster.
    
    With respect to contraction, each \varUIMA~pipeline must be prepared to
    process work items that may have been partially processed previously.
   
    Pipelines themselves may comprise one or more duplicate threads, such that each
    pipeline can simultaneously process multiple work items.
    The number of pipelines and threads per pipeline are configurable per \varJob.
   
    \subsection{Performance}  
    
    For the distributed environment, \varDUCC~relies upon a \varNetworkFileSystem~(\varNFS)
    for file access to work items.
    High performance is achieved through \varNFS~data sharing and (via \varActiveMQ) the passing of
    data-handles that are utilized by the "embarrassingly parallel" pipelines.
    
    \section{\varReservations}
    
    To help support Jobs, \varDUCC~provides facilities for \varReservations~of two types: 
    Managed and Unmanaged. \varReservations, once allocated, are preserved until 
    canceled. 
    
    \varManagedReservations~(MRs) comprise "arbitrary" processes, for example Java
    programs, c-programs, bash shells, etc.
    
    \varUnmanagedReservations~(URs) comprise a resource that can be utilized for any 
    purpose, subject to the limitations of the assigned \varShare~or \varShares.
            
    \section{\varServices}
       
    To help support \varJobs, \varDUCC~provides facilities for \varServices~of two types: 
    \varUIMA~and \varPingOnly. \varServices~can be predefined in a registry, and 
    \varJobs~can declare dependency on one or more of them.
    
    \varServices~can be shared by my multiple \varJobs~or can be tied to just one.
    \varServices~can be started at \varDUCC-boot time or at \varService-definition time or
    at \varJob~launch time.
    
    \varServices~can be expanded and contracted by command or on-demand.  
    \varServices~can be stopped by command or due to absence of demand.
    
    \varServices~nominally exists for reasons of efficiency due to high start-up costs
    or high resource consumption.  Benefits of cost amortization are realized by sharing
    \varServices~amongst a collection of \varJobs~rather than employing a private copy 
    for each.
    
    The lifecycle of each \varUIMA~\varService~is managed by \varDUCC, which is not the
    case for \varPingOnly~\varServices.  However, each comprises a "pinger" which 
    adheres to a standard interface and provides health and statistical information.
    
    \section{Management} 
    
    The \varDUCC~system employs several management techniques to fairly apportion
    resources.
    
    \subsection{Memory Shares} 
    
    The \varDUCC~system partitions the entire set of available resources comprising
    \varNodesMachinesComputers~into \varShares.
    
    Partitioning of the available \varNodesMachinesComputers~into \varShares~facilitates
    multitenancy amongst a collection of \varDUCC-managed user applications consisting 
    of \varUIMA~pipelines.
    
    One or more \varShares~are allocated and sub-partitioned into \varJdShares.
    
    Users submit \varJobs~to the \varDUCC~system specifying a requisite memory size.
    Each \varJob~is allocated one \varJdShare~and, based upon user specified
    memory size, one or more \varShares. 
    Likewise, users submit \varReservations~and \varServices~also comprising memory size
    information.  These are assigned \varShares~only.
    
    New \varJobs, \varReservations~and \varServices~may only enter the system when
    there are sufficient unallocated \varShares~available. To make room for newly arriving
    submissions, the \varResourceManager~may preempt use of already previously
    assigned \varShares~for re-assignment.
    
    \subsection{\varLinuxControlGroups} 
    
    If available, \varDUCC~employs \varLinuxControlGroups~to enforce limits on
    deployed applications. Exceeding limits penalizes only the offender.
    For example, if a user application exceeds its memory \varShare~size then it is forced
    to swap while other co-resident applications remain unaffected.
    
    \subsection{Preemption} 
        
    Preemption is employed by \varDUCC~to re-apportion \varShares~when new work is submitted.
    
    For example, presume a simple \varDUCC~system with just one preemptable scheduling class
    and resources comprising 11 \varShares. Further, suppose that 1 \varShare~is allocated 
    for partitioning into \varJdShares.
    When the Job \#1 is submitted it is entitled to all remaining 10 shares.  
    When Job \#2 arrives, each job is entitled to only 5 shares.
    Thus, 5 \varShares~from Job \#1 are preempted and reassigned to Job \#2.
    
\chapter{System Organization}

    \section{Single System Image}
    
    \varDUCC~runs on \varLinux. It can be run on a single system in simulation-mode
    or on a cluster (two or more machines). For clusters, \varDUCC~replies upon 
    these requirements:
    
    \begin{itemize}
      \item common userids across the cluster
      
      Each userid must have the same definition on all machines participating
      in the \varDUCC~cluster.
      
      \item a shared filesystem for user and \varDUCC~data across the cluster
      
      Each machine shares a filesystem (commonly provided by NFS) with all 
      machines participating in the \varDUCC~cluster.
      
    \end{itemize} 
    
    \section{Communications}
    
    \varDUCC~comprises a collection of singleton and distributed daemons that need
    to coordinate activities.  This coordination is accomplished via messaging.
    
    The system is fault tolerant with respect to lost messages, since
    publications occur at regular intervals and each message encapsulates
    the current and/or desired state for the target audience.
    As such, actions may be be delayed but will be carried out as soon as the
    next message arrives.
    
    \section{Daemons}
    
    \varDUCC~is implemented through a collection of configurable singleton 
    and distributed daemons.
    
    \input{part5/ducc-pops-component-orchestrator.tex}
    \input{part5/ducc-pops-component-resource-manager.tex}
    \input{part5/ducc-pops-component-services-manager.tex}
    \input{part5/ducc-pops-component-process-manager.tex}
    \input{part5/ducc-pops-component-agent.tex}
    \input{part5/ducc-pops-component-job-driver.tex}
    \input{part5/ducc-pops-component-user-interface.tex}
    \input{part5/ducc-pops-component-webserver.tex}

    \section{Interfaces}
    
    Interfaces description.
    
\chapter{Runtime}
    
    \section{State Machines}
    
    \subsection{Job State Machine}    
       
        \begin{table}[H]
        \caption{Job State Machine}
        \begin{tabular}{{l}{l}{l}{l}}
        Id      & Name                      & Next           & Description \\
        \hline
        1       & Received                  &  2, 7          & Job has been vetted, persisted, and assigned unique Id \\
        2       & WaitingForDriver          &  3, 4, 7       & Process Manager is launching Job Driver \\         
        3       & WaitingForServices        &  4, 7          & Service Manager is checking/starting service dependencies for Job \\
        4       & WaitingForResources       &  5, 7          & Scheduler is assigning resources to Job \\
        5       & Initializing              &  6, 7          & Process Agents are initializing pipelines \\
        6       & Running                   &  7             & At least one Process Agent has reported process initialization complete \\
        7       & Completing                &  8             & Job processing is completing \\
        8       & Completed                 &                & Job processing is completed
        \end{tabular}
        \end{table}
    
    \subsection{Service State Machine}   
    
        \begin{table}[H]
        \caption{Service State Machine}
        \begin{tabular}{{l}{l}{l}{l}}
        Id      &Name                       & Next           & Description \\
        \hline
        1       & Received                  &  2, 3, 6       & Service has been vetted, persisted, and assigned unique Id \\
        2       & WaitingForServices        &  3, 6          & Service Manager is checking/starting service dependencies for Service \\
        3       & WaitingForResources       &  4, 6          & Scheduler is assigning resources to Service \\
        4       & Initializing              &  5, 6          & Process Agents are initializing pipelines \\
        5       & Running                   &  6             & At least one Process Agent has reported process initialization complete \\
        6       & Completing                &  7             & Service processing is completing \\
        7       & Completed                 &                & Service processing is completed
        \end{tabular}
        \end{table}
    
    \subsection{Reservation State Machines}     
    
        \begin{table}[H]
        \caption{Unmanaged Reservation State Machine}
        \begin{tabular}{{l}{l}{l}{l}}
        Id      &Name                       & Next           & Description \\
        \hline
        1       & Received                  &  2, 3          & Reservation has been vetted, persisted, and assigned unique Id \\
        2       & Assigned                  &  3             & \varShares~are assigned \\
        3       & Completed                 &                & \varShares~not assigned  
        \end{tabular}
        \end{table}
     
        \begin{table}[H]
        \caption{Managed Reservation State Machine}
        \begin{tabular}{{l}{l}{l}{l}}
        Id      &Name                       & Next           & Description \\
        \hline
        1       & Received                  &  2, 3, 5       & Reservation has been vetted, persisted, and assigned unique Id \\
        2       & WaitingForServices        &  3, 5          & Service Manager is checking/starting service dependencies for Reservation \\
        3       & WaitingForResources       &  4, 5          & Scheduler is assigning resources to Reservation \\
        4       & Running                   &  5             & Process Agent has reported program launched \\
        5       & Completing                &  6             & Reservation processing is completing \\
        6       & Completed                 &                & Reservation processing is completed
        \end{tabular}
        \end{table}
         
    \section{Dependencies}
    
    \section{Scheduling}
    
    \section{Monitoring and Control}
    
    \subsection{Automatic} 
    
    \subsection{Manual} 
        
    \section{Logging}
        
    \subsection{System} 
    
    \subsection{User} 
    
    \section{Recovery}
  
    \subsection{System} 
    
    \subsection{User}