// Licensed to the Apache Software Foundation (ASF) under one or more
// contributor license agreements. See the NOTICE file distributed with
// this work for additional information regarding copyright ownership.
// The ASF licenses this file to You under the Apache License, Version 2.0
// (the "License"); you may not use this file except in compliance with
// the License. You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

== Implementation Details

=== Fault-Tolerant Executor (FATE)

Accumulo must implement a number of distributed, multi-step operations to support
the client API. Creating a new table is a simple example of an atomic client call
which requires multiple steps in the implementation: get a unique table ID, configure
default table permissions, populate information in ZooKeeper to record the table's
existence, create directories in HDFS for the table's data, etc. Implementing these
steps in a way that is tolerant to node failure and other concurrent operations is
very difficult to achieve. Accumulo includes a Fault-Tolerant Executor (FATE) which
is widely used server-side to implement the client API safely and correctly.

FATE is the implementation detail which ensures that tables in creation when the
Master dies will be successfully created when another Master process is started.
This alleviates the need for any external tools to correct some bad state -- Accumulo can
undo the failure and self-heal without any external intervention.

=== Overview

FATE consists of two primary components: a repeatable, persisted operation (REPO), a storage
layer for REPOs and an execution system to run REPOs. Accumulo uses ZooKeeper as the storage
layer for FATE and the Accumulo Master acts as the execution system to run REPOs.

The important characteristic of REPOs are that they implemented in a way that is idempotent:
every operation must be able to undo or replay a partial execution of itself. Requiring the
implementation of the operation to support this functional greatly simplifies the execution
of these operations. This property is also what guarantees safety in light of failure conditions.

=== Administration

Sometimes, it is useful to inspect the current FATE operations, both pending and executing.
For example, a command that is not completing could be blocked on the execution of another
operation. Accumulo provides an Accumulo shell command to interact with fate.

The +fate+ shell command accepts a number of arguments for different functionality:
+list+/+print+, +fail+, +delete+, +dump+.

==== List/Print

Without any additional arguments, this command will print all operations that still exist in
the FATE store (ZooKeeper). This will include active, pending, and completed operations (completed
operations are lazily removed from the store). Each operation includes a unique "transaction ID", the
state of the operation (e.g. +NEW+, +IN_PROGRESS+, +FAILED+), any locks the
transaction actively holds and any locks it is waiting to acquire.

This option can also accept transaction IDs which will restrict the list of transactions shown.

==== Fail

This command can be used to manually fail a FATE transaction and requires a transaction ID
as an argument. Failing an operation is not a normal procedure and should only be performed
by an administrator who understands the implications of why they are failing the operation.

==== Delete

This command requires a transaction ID and will delete any locks that the transaction
holds. Like the fail command, this command should only be used in extreme circumstances
by an administrator that understands the implications of the command they are about to
invoke. It is not normal to invoke this command.

==== Dump

This command accepts zero more transaction IDs.  If given no transaction IDs,
it will dump all active transactions.  A FATE operations is compromised as a
sequence of REPOs.  In order to start a FATE transaction, a REPO is pushed onto
a per transaction REPO stack.  The top of the stack always contains the next
REPO the FATE transaction should execute.  When a REPO is successful it may
return another REPO which is pushed on the stack.  The +dump+ command will
print all of the REPOs on each transactions stack.  The REPOs are serialized to
JSON in order to make them human readable.