% 
% Licensed to the Apache Software Foundation (ASF) under one
% or more contributor license agreements.  See the NOTICE file
% distributed with this work for additional information
% regarding copyright ownership.  The ASF licenses this file
% to you under the Apache License, Version 2.0 (the
% "License"); you may not use this file except in compliance
% with the License.  You may obtain a copy of the License at
% 
%   http://www.apache.org/licenses/LICENSE-2.0
% 
% Unless required by applicable law or agreed to in writing,
% software distributed under the License is distributed on an
% "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
% KIND, either express or implied.  See the License for the
% specific language governing permissions and limitations
% under the License.
% 

% \section{DUCC Service Manager}
    This section describes the architecture and internal structure of the
    DUCC Service Manager, referred to as the ``SM''.

\section{Introduction}
    The SM function is to insure that any services needed by
    DUCC jobs are running and functional at the time they are needed by
    jobs.  Previous to the incarnation of the SM it was necessary for
    users to manually invoke the processes implementing their services.  If
    these processes were to crash, jobs dependent on them would stop until
    some human was able to restart the service.  If the operating system,
    or batch system supporting the jobs (DUCC, in our case) was to
    be restarted, users would again have to manually start the services.

    By ``registering'' a service with the SM, a user can trust DUCC to
    keep the service alive and functional across all manner of faults and
    system restarts.  As well, the SM has a mechanism for ``testing'' a
    service to determine if it is operational, and to inform the DUCC
    Web Server when it is not.  

    If a user submits a job that declares a dependency on a service, the SM
    is able to start the service as needed, and is able to stop the service
    when no longer needed, freeing resources.

    In essence, the SM can be thought of as a ``proxy user'' dedicated
    to insuring that services are always available when needed.

\section{Architectural Overview}

    Figure ~\ref{fig:sm-structure} below shows the high-level object,
    threading, and process structure of SM and should be referenced
    while reading this document.
    
    The SM can be pictured as being composed of four major parts:
    \begin{enumerate}
      \item Initialization and interaction with external components.
        External components include user requests and  other DUCC components such as
        the Orchestrator.
      \item A ``Service Instance'' manager.  This part resolves
        dependencies on services, starts and stops service instances
        according to the needs of jobs and the policies declared in
        the service registries, and handles the service instance
        lifetimes.
      \item A ``Service Health'' manager.  This part continually
        ``tests'' services to determine whether they are 
        functional.  This is referred to as the ``pinger'' and the
        test is known as a ``ping''.
      \item A ``CLI Handler'' which reacts to requests from users.
    \end{enumerate}

    \begin{figure}[H]
      \centering
      \includegraphics[width=5.5in]{images/ducc-internals/sm-structure.png}
      \caption{Service Manager Structure}
      \label{fig:sm-structure}
    \end{figure}

   The terminology around Services can be confusing.  We review the ideas here.

   There are three ``countable'' entities involved in services.  
   \begin{description}
     \item[Service Registration] When a service is ``registered'' the Service Manager assigns
       a new, unique {\em Registration ID} to the registration.  This ID is associated with, and
       remains with, the service throughout its lifetime and beyond when it is archived.

     \item[Service Instance] When the Service Manager starts a service it issues a series of
       ``submit'' orders to the Orchestrator, one for each {\em Service Instance}.  All these 
       instances are associated with the {\em Service Registration}.  The orchestrator assigns
       a unique ID to each service instance, which is also permanently associated with that entity.
       
       This is analogous to a {\em job}, but with a single allocation.  The SM organizes
       multiple {\em Service Instances} under a single {\em Service Registration}.
       
     \item[Share Id] When an instance is submitted to the Orchestrator, the Orchestrator
       ``submits'' a request to the Resource Manager to find resources.  Each instance is
       treated independently by the RM.  When a resource is found for the instance, the RM
       assigns a {\em Share ID} to the allocation. 
       
       This is analagous to a {\em job's} process or ``PE''.
    \end{description}

    The {\em Service Registration} ID appears on the Services' page of the webserver.  The {\em
      Service Instance} ID and the {\em Share Id} appear in the ID column of the service details in
    the DUCC webserver.  For example, this ID: {\tt 289661.34094} indicates {\em Service Instance}
    ID {\tt 289661} and {\em Share Id} {\tt 34094}.

\section{Initialization and Interaction with DUCC}
   The component responsible for initialization and external interaction is
   implemented in the source file {\em ServiceManagerComponent.java}.  This
   straightforward bit of code performs the following functions, details of which
   are easy to understand by reading the source code itself.  

   \begin{description}
     \item[Initialization] This consists of the methods init() and
       start(). The DUCC framework instantiates ServiceManagerCompoent and calls
       its start() method.  This initializes various structures from
       {\em ducc.properties}, initializes the database connection, and
       initializes the two main threads:
       \begin{enumerate}
         \item The SM proper, {\em ServiceManagerComponent}, which
           fires its {\em run()} method which in turn calls {\em init()}.
           \item The Service Instance manager, implemented in {\em ServiceHandler.java}.
       \end{enumerate}
       The {\em init} method reads all registrations from its state repository and
       passes them to {\em register()} (using the same code path as the {\em register} CLI),
       to establish them in a memory map and possibly initialize them.


       \item[Interaction With DUCC] There are three primary interactions to be aware of:
         \begin{enumerate}
           \item Incoming Orchestrator publications.  This arrives on an external
             communication thread  and passed to
             the method {\em orchestratorStateArrives} which, if it accepts the
             publication, saves the incoming publication and issues a {\em notify()} to 
             the main {\em ServiceManagerComponent} thread to allow processing of the state.  

             The method {\em processIncoming} is then called which does standard DUCC
             state-differencing and passes updates to the {\em Instance Management} code
             in {\em ServiceHandler}.

           \item CLI requests.  These are passed via the usual DUCC event handlers to specific
             second-level handlers, one for each type of CLI request (e.g. {\em register()}).  Each of these
             second-level handlers is responsible for these actions:
             \begin{description}
               \item[User validation.]  Insure the caller of the CLI is {\em authenticated}, i.e.
                 is the user he claims to be.
               \item[Ducc is running.]  The DUCC Orchestrator must be actively publishing state
                 before SM is allowed to interact with users.                 
             \end{description}
             
             If these simple tests are passed, the request is passed to the Instance Management
             code in {\em ServiceHandler} to check for authorization (i.e. is this user allowed
             to perform this action against this service).  

           \item Outgoing state publications.  Outgoing state is a simple map, one entry per
             job (a ``job'' for SM is any unit of work in the system: UIMA-AS job, Service instance,
             Reservation, AP).  The entry contains the state of the job relative to any
             services it depends on, which is interpreted by the Orchestrator and Web Server.

         \end{enumerate}

         Note that Orchestrator publications and CLI requests may be ignored under these two conditions:       
         \begin{enumerate}
         \item SM initialization is not complete.  Completion is flagged as the last
           action of {\em init()}.
         \item RM has not yet assigned the JD node.  The incoming OR publication includes a flag to
           indicate whether the JD node is assigned.  We have to wait here because we do not want
           the SM to process work or initialize any services until it is confirmed that the system
           is fully initialized.  Note that this minimizes the occurrence of errors and simplifies
           error management because you can all errors occurred in a fully initialized environment.
         \end{enumerate}
         
     \end{description}
   

\section{Service Instance Management: ServiceHandler and ServiceSet}
    After the differencing engine has determined the various work events that have
    occurred, the Service Instance Management code examines each event
    and acts upon it.  

\subsection{Operational Overview}
    The code described below runs in a thread separate from the main thread
    described in the previous section.  Incoming events are placed on lists
    segregated by function (a list for new Jobs, a list for updated Jobs, etc).  As soon
    as all incoming events are placed on these lists the {\em Service Instance Management}
    thread is {\em notified}.  The {\em Service Instance Management} thread sets a lock,
    drains the lists into internal structures, and releases the lock.  This segregates the
    the actions of the {\em ServiceManagerComponent} from {\em Service Instance Management}.

    As incoming events are acted upon, a summary of the service state for all
    incoming work is built up.  After all events are processed, the {\em ServiceManagerComponent} 
    (previous section) is notified and the state publication is sent to the Orchestrator.

    There are two primary components involved in {\em Service Instance Management:}
    \begin{description}
      \item[ServiceHandler.java] This is a singleton object which runs in its own 
        thread sepparate from the {\em ServiceManagerComponent}.  It fields the
        updates from the Orchestrator and CLI, resolves Service dependencies, and signals
        the {\em ServiceSet} for each affected service so appropriate action can be taken.
        It maintains all the records of registered services and service-dependent jobs
        in an inner class {\em ServiceStateHandler}.
      \item[ServiceSet.java] There is one {\em ServiceSet} for every registered service.  It is
        instantiated on receipt of a registration and destroyed only when a service is unregistered.
        It is responsible for submitting service instances to the Orchestrator, reacting to state
        changes of the Service Instances, enforcing management policies ({\em reference start}, {\em
          autostart}, {\em manual start}), and fielding the data from the service Pinger.
    \end{description}
      
\subsection{ServiceHandler.java}
    The work-related events, fielded by {\em ServiceHandler}, are described below.
    These events can be placed into two broad categories:
    \begin{enumerate}
      \item Events relating to work that requires services, usually UIMA-AS jobs
      \item Events relating to service instances for registered services.  
    \end{enumerate}
    
    Within each category are three types of interesting events.
    \begin{enumerate}
      \item A new job or service instance has entered the system.  This is essentially a REQUEST
        from the Orchestrator, asking if all necessary services are available.  No work
        has been started in the system, and will not be until SM responds ``services available''
        to the request.

        NOTE that the Orchestrator has a ``fast-path'' for work that has no service dependencies, in
        that it does not wait for the SM to respond regarding such work.  SM does in fact respond,
        but after the fact, and the response is not used.

      \item An existing job or service instance has changed state.  This is work that the 
        Orchestrator has started: there are physical processes either started, or in the act
        of starting and their states may be evolving.

      \item An existing job or service instance has terminated.
    \end{enumerate}
    
    While technically any type of DUCC work can be dependent on a service, by far the most common
    is UIMA-AS Jobs and Service instances.  The SM must treat work which IS a Service Instance
    a little differently from all other work (because all work is potentially depdenent on the
    state of Service Instances). Below we will use the term ``Job'' to refer to any
    kind of work that is not a service instance.

    NOTE: for simplicity, the descriptions in this section use the term ``services available'' to refer
    to the single state {\em Available} which indicates a service is running and is successfully
    pinging, and  ``services unavailable'' to refer to all other states.  The complete set of
    states is encoded in the class
\begin{verbatim}
org.apache.uima.ducc.transport.event.sm.IService.java
\end{verbatim}
    in the enum {\tt ServiceState} as shown below.
\begin{verbatim}
    public enum ServiceState 
    {
      Pending,        // Work is waiting on at least one service to start
      Waiting,        // A job is waiting on at least one service to ping
      Starting,       // Instance is started, but not yet to Initializing
      Initializing,   // A job is waiting on at least one service to initialize
      Available,      // All services for this job are active and pinging, or else
                      //     no services are needed for the job
      NotAvailable,   // SM to OR only: reference to a non-existent service 
      Stopped,        // The service is not started
      Stopping,       // Service is told to stop but it takes a while
      Undefined,      // Catch-all, means basically "who cares"
      ;
    }
\end{verbatim}

    \paragraph{Service Events}. Service events are {\em processed in the order shown
    below.} The order is important because overall service state is advancing through the
    first three events.
    
    \begin{description}
      \item[A new Service Instance has arrived.]  The associated {\em ServiceSet}
        is found and signalled.  If the associated {\em ServiceSet} cannot be found this
        is considered a ``rogue'' service instance and is ignored.  This occurs if
        the incoming process cannot be matched with a registered service; for example, if
        the {\em DuccServiceSubmit} CLI is called outside of SM.  Usually this is an
        error condition but is not considered fatal by SM.

        If this service is dependent on other services, the {\em ServiceSet}s for those
        other services are fetched and signalled, which may in turn cause additional
        service instances to be submitted.  If all of these other services are Running,
        the new service is marked ``services available'' and the Orchestrator will 
        physically start the new instance.  Otherwise it is marked ``services unavailable''
        and will not be allowed to start until it's own service dependencies are running.

        Note how this implements a sort of ``domino'' effect for starting services.  Suppose
        you have two services, A dependent on B, and a job dependent on A,
        with none of these running.  When job A arrives at SM it is marked ``services unavailable'' and
        service A is submitted.  When service A arrives from Orchestrator it is marked ``services unavailable'' and
        service B is submitted.  When service B arrives from Orchestrator it is marked ``services available'' so 
        that Orchestrator may start it.  When it starts, service A is marked ``services available'' and is
        started by Orchestrator.  When service A starts, the job is finally marked ``services available'' and
        is allowed to start.

        As mentioned above, if the services is NOT dependent on other services, the Orchestrator
        will fast-path its start, and SM will mark it ``services available''.
     
      \item[An existing Service Instance has arrived.]  The {\em ServiceSet} is
        fetched and signalled with the state of the incoming instance.  This may cause
        the Service State to be updated, for example from Initializing to Running.  Pingers
        may be started as needed.

        Note that, because this is processed BEFORE jobs are processed, job state may be
        updated from ``services available'' to ``services unavailable'' (or vice-versa).

      \item[A Service Instance has exited.]  Some service instance for some managed
        service has exited for some reason.

        The associated {\em ServiceSet} is found and signalled.  The {\em ServiceSet}
        takes appropriate action, restarting the instance if the exit
        was unexpected, or perhaps simply updating its records if the service is
        being stopped or the number of instances reduced.

      \item[A new job has arrived.]  
        The declared service dependencies are parsed and the {\em ServiceSet}
        object for each such service is signalled.   The {\em ServiceSet} increments the reference count for
        the service and, in the case of {\em reference-started}
        services, starts some number of instances (using the ``hidden'' CLI utility
        {\em DuccServiceSubmit}.

        Based in the current state of the service, the job is flagged with ``services available'' or
        ``services unavailable''.  The job is marked ``services available'' if-and-only-if all
        its declared services are started, in running state, and being successfully {\em tested} (``pinged'').

      \item[An existing job has arrived.]  Work that is not new may need its service state reevaluated.
        The declared service dependencies are parsed and the associated {\em ServiceSet}
        objects are fetched.  An analysis of the combined work states and service
        states is done.  The job is flagged ``services available'' or ``services unavailable''.

        Note that a job's service-state can change from ``services available'' to ``services
        unavailable'' if a service fails for some reason.

      \item[A job has exited.]  A job which might be dependent
        on a service has left the system. 

        The ServiceHandler examines all the declared services for the work.  For each
        dependency, if the service is started, the ServiceSet for the service is 
        signalled.  Each affected {\em ServiceSet} decrements its reference count.  If the
        count goes to zero, and if this is
        a reference-started service, the {\em ServiceSet} stops all instances.
          
    \end{description}
    
\subsection{ServiceSet.java}
{\em ServiceSet} is responsible for the care-and-feeding of the set of objects, threads, and
processes used to manage an individual service.  There is one {\em ServiceSet} instantiated for
every registered service.  {\em ServiceSet} does NOT run in its own thread.  Its methods are always
executed either on the thread of the {\em ServiceHandler} or on a {\em pinger} thread. It is
responsible for maintaining the correct number of running instances, fielding pings, and updating
the state repository (as of DUCC 2.1.0, a database) for a single service.

   The primary functions include:
   \begin{description}
     \item[Enforce Autostart]  If the service is registered for autostart, insure sufficient instances are started
       and start new ones if needed.  This is called at the end of each update cycle from the {\em ServiceHandler}.

     \item[Manage references] Maintain a reference count to reflect all work that is
       referencing the service.  If the service is ``reference started'' this can trigger
       the start of new instances and the shutdown of existing instances.  This count is
       always maintained so if the administrator changes the start-up policy for the service,
       the reference count is already correct and is used.

    \item[Start-up and hot start] On hot-start, the method {\em bootInstances()} is called to
      synchronize each service set with the incoming orchestrator publications, followed by
      bootComplete() to finalize bookkeeping and update the meta state data.

    \item[Field pings] When the Orchestrator state indicates that at least one instance for the
      service is in Running state, the {\em ServiceSet} starts a pinger.  After every ping,
      the pingers call {\em signalReblance()} to respond to the ping and enforce any
      potential state changes as a result of the ping, including starting and stopping 
      specific instances.

    \item[Sequence Instance Startup] The {\em ServiceSet} also sequences instance start-up so that
      in general, there is only a single instance starting at  time.  This is managed in the
      method {\em needNextStart()};

      Instances are sequenced up for several reasons:
      \begin{itemize}
        \item Avoid flooding the system with start requests during boot.
        \item Avoid start/fail loops with faulty services.
      \end{itemize}
      
    \item[Respond to Instance State Change]  The {\em ServiceHandler} invokes the {\em ServiceSet}
      method {\em signalUpdate} on every Orchestrator update.  This method examines the entire
      state of the service and coordinates any actions which may be triggered by state change
      of a service instance.

    \item[State Accumulation]  The state of a service is dependent on the cumulative states of
      all of its instances PLUS the state if its pinger.  The method {\em cumulativeJobState()}
      is responsible for state accumulation.

      The design point is that states are associated with an ordinal.  The larger the ordinal,
      the ``closer to functional'' a service is.  The lower the ordinal, the ``farther from functional''
      the service is.  State accumulation walks the states of all relevant components maintaining
      the {\em maximum} state encountered.  This {\em maximum} is considered to be the state of the
      service overall.  The method {\em translateJobState()} is used to assign the ordinal
      based on the state of the service instances.

    \item[State Management] There is a very simple state machine managed by two methods,
      {\em signal()} and {\em setState()}.  The details can be found by examining these methods.
      The design point is this:  most actions in the SM are considered {\em idempotent}.  Thus,
      regardless of the outcome of state change (or lack of state change), these actions are
      ALWAYS called; for example, ``start the pinger''.  The methods implementing the actions
      are responsible to determine whether the current situation is compatible with the requested
      action.  For example, if the method {\em startPingThread()} is called, that method must
      check to see if the ping thread is already running, and not start a new thread if so.

      This design makes the state machine extremely simple and easy to maintain.

      \item[Lingering Stop] {\em Lingering stop} occurs in a reference-started service when the
        final reference has exited.  It is implemented by a Java TimerTask, {\em LingerTask}.
        If the timer ``pops'', this task stops the service.  If a new reference arrives
        before the timer ``pops'', the task is canceled.

   \end{description}

\subsection{ServiceInstance.java}
    The {\em ServiceInstance} object is a simple helper whose responsibility it is to start
    a new instance.  It spawns a {\em DuccServiceSubmit} process as the user via {\em ducc\_ling} with a pointer to the
    registration properties, and scrapes the output in the response to get the Orchestrator-assigned
    ID.  The ID and any error messages are returned to the calling {\em ServiceSet}.

    This mechanism is used to manage ping-only services as well.  When a ping-only service is started,
    a subclass of ServiceInstance is started.  This {\em PingOnlyServiceInstance} simulates the start
    of an actual instance.  It maintains an internal thread that invokes {\em ServiceSet.signal()} to simulate
    the {\em ServiceHandler}'s regular state updates.  It is essentially a ``service proxy'' for the 
    non-DUCC-handled service that is being pinged, eliminating the need for most special cases in the
    handling of ping-only services.

\section{Service Health Management: Ping support}
    When the {\em ServiceSet} detects that at least one instance has achieved {\em Running} state,
    it calls the method {\em startPingThread()}.

    {\em startPingThread()} starts a thread dedicated to managing the pinger, with the class
    {\em PingDriver} running as its (logical) ``main''.  The {\em PingDriver} reaches back into its
    {\em ServiceSet} for the parameters needed to manage the pinger (Java class, ping interval,
    etc).

    The {\em PingDriver} implements two cases:
    \begin{enumerate}
      \item Internal pingers
      \item External pingers
    \end{enumerate}
    
    In both cases, the {\em PingDriver} object creates an object from {\em Ping.java}, loads
    the object with a {\em Map} of user and DUCC-supplied parameters, and passes the {\em Ping} to
    the ping implementation.  A timer is set and a response is waited for.  

    In both cases the ping mechanism collects ping parameters and passes them to the {\em AServicePing}
    object which was extended to create the pinger.  The ping mechanism then calls various
    methods on {\em AServicePing} to extract its state, constructs a {\em Pong} object, and uses
    this to communicate with the SM by invoking {\em ServiceSet.signalRebalance()}.  The details
    of how this occurs is different for internal and external pingers in the {\em PingDriver}.
    
    \subsection{Internal Pingers}

    If the pinger is registered as an ``Internal'' pinger, a Java ClassLoader is invoked to load
    the pinger's registered class. A timer loop is established to invoke the pinger.  On
    each invocation, the {\em PingDriver} directly invokes the methods on {\em AServicePing} 
    required to invoke the pinger and retrieve its state.  A {\em Pong} object is created
    from the state and passed to common code in the method {\em handleResponse()}.

    \subsection{External Pingers}

    If the pinger is registered as an ``External'' pinger, a ServerSocket is started and its listen
    port acquired. An instance of the class {\em ServicePingMain} is spawned under the identity of
    the owning userid of the service via {\em ducc\_ling}, passing the listen port and appropriate
    arguments needed to start the user's pinger. The stdout and stderr of the resultant process is
    captured and written to the SM log and the ping loop (describe below) is started.
    
    The {\em ServicePingMain} object receives a set of start-up parameters including the
    name of the user's ping class, classpath, start-up parameters, and endpoint.  The user's
    pinger, extending {\em AServicePing} is instantiated by {\em ServicePingMain} using a
    ClassLoader.  {\em ServicePingMain} then goes into
    an infinite, untimed loop waiting for the {\em Ping} requests from the {\em PingDriver}.  When
    a request arrives, the parameters in the request are gathered and sent to the user's pinger
    by calling methods on {\em AServicePing} as is done for internal pingers.
    The response is received and returned to {\em ServiceSet} in a {\em Pong} object.
    
    The embedded class {\em PingDriver.PingThread} is used to implement the protocol between SM and
    the external pinger.   The wire protocol between SM and {\em ServicePingMain} is quite simple.

    On the SM side:
    \begin{itemize}
      \item SM starts a ServerSocket and then issues an {\em accept()} waiting for the external pinger to connect.
      \item Once the pinger connects, enter a loop until terminated by SM:
        \begin{itemize}
          \item Write a new {\em Ping} object to the socket stream.
          \item Read a new {\em Pong} object from the socket stream.
        \end{itemize}
    \end{itemize}
    
    If SM terminates the pinger, the {\em Ping} object includes a flag that is read by
    {\em ServicePingMain} causing it to shutdown the user's pinger and exit.

    On the external pinger's side, with {\em ServicePingMain} started by {\em ducc\_ling} as
    the owner of the service:
    \begin{itemize}
      \item Read the command-line parameters and then:
        \begin{itemize}
          \item Connect to the SM on it's listen socket.
          \item Instantiate the user's ping class using a classloader.
        \end{itemize}
      \item Enter a loop:
        \begin{itemize}
          \item Read the next Ping
          \item If the Ping has the {\em exit} flag set, call the pinger's {\em stop()} method, close the
            input and output streams, and exit.
          \item Otherwise invoke the pinger's base methods defined in {\em AServicePing}.
          \item Construct a {\em Pong} object and write it to the SM's socket.
        \end{itemize}
      \end{itemize}
    

    \section{CLI Management}
    The CLI management code is relatively simple but has a few details worth mentioning.

    {\em ServiceManagerComponent} receives the incoming CLI call, insures the user identified
    in the incoming packet is the one who submitted it, that the DUCC Orchestrator processes
    is functioning, and passes the incoming packet to {\em ServiceHandler}.  Each different
    CLI function is passed to the appropriate second-level handler in {\em ServiceHandler}; for
    example, a registration request is passed to {\em ServiceHandler.register()}.

    Secondary vetting of the incoming request is then performed: 
    \begin{itemize}
      \item Check to see if the service exists.  If so, and it's a registration then fail with a
        ``duplicate services'' message.  If it is NOT a registration and it does not exist, fail with a
        ``service not found'' message.
      \item Insure the caller is authorized for the desired action (e.g. is the caller an
        admin or owner for the service).
      \item If the request can be performed immediately, do so and return confirmation.  These requests are
        \begin{itemize}
          \item Register
          \item Query
          \item Enable
          \item Disable
          \item Ignore references
          \item Observe references
        \end{itemize}
      \item If the request can take an arbitrarily long time, then build a deferred CLI object
        and enqueue it.  This deferred object is implemented in {\em ApiHandler.java}.  Return
        confirmation to the user that the request is accepted.  These requests are
        \begin{itemize}
          \item Start
          \item Stop
          \item Modify
          \item Unregister
        \end{itemize}
        
    \end{itemize}
      
    Deferred requests are dequeued and executed one at a time.  The user is not directly
    informed of this execution as it may complete at an arbitrary time in the future.  For
    example, stopping or unregistering a service both involve an elaborate sequence of
    actions to stop the processes, deallocate the space, and update the SM records.
    Starting a service is also quite elaborate and additionally, service starts are
    sequenced so they are submitted one-at-a-time; services with many instances can take
    a significant time to fully start.

    These requests are synchronized with other SM activities to minimize race conditions by
    handling them in the same execution thread in {\em ServiceHandler} that is handling incoming Orchestrator
    publications.  Deferred requests are implemented in two methods each:
    \begin{enumerate}
      \item An ``immediate'' method which vets the CLI parameters, constructs, and enqueues the deferred request:
        \begin{itemize}
          \item ServiceHandler.start()
          \item ServiceHandler.stop()  
          \item ServiceHandler.modify()
          \item ServiceHandler.unregister() 
          \end{itemize}                      
      \item A ``deferred'' method, invoked from the deferred request:
        \begin{itemize}
          \item ServiceHandler.doStart();
          \item ServiceHandler.doStop();
          \item ServiceHandler.doModify();
          \item ServiceHandler.doUnregister();
        \end{itemize}
        
    \end{enumerate}