docs/source/setup.rst

Setting up Apache Warble
========================

.. toctree::
   :maxdepth: 2
   :caption: Contents:


****************************
Understanding the Components
****************************

Warble currently consists of two major components:

The Warble Master Server (warble-server)
   This is the main database and UI Server. It serves as the hub for the
   nodes/agents to connect to, and provides the overall management of
   hosts, tests, as well as the visualizations and API end points.
   
The Warble Node Applications (warble-node)
   This is a daemon with a collection of test classes used to test 
   external hosts for various services and/or response values. Nodes
   send results back to the master, which then processes and responds
   accordingly (for instance, in the case of downtime).

A third major component, the Warble Agent Applications, are being 
worked on, but is not completed.

**********************
Component Requirements
**********************

################
Server Component
################

The main Warble Server is a hub for nodes/agents and tests, and as such, is
generally speaking only needed on one machine. It is recommended that, for larger
instances of warble, you place the application on a machine or VM with
sufficient resources to handle the database load and memory requirements.

We will be working towards a multi-master setup option, but that is 
currently not available.

As a rule of thumb, the Server does not require a lot of disk space
(enough to hold the compiled database and timeseries), but it does require CPU and RAM.
The nodes/agents require virtually no disk space, as all test results are sent 
to the master server for storage.

#################
Node Component
#################

The node component can either consist of one instance, or be spread
out across multiple machines for a distributed test coverage. 
Nodes will auto-adjust the test speed to match the number of CPU cores available to it; 
a node with two cores available will run up to 256 simultaneous jobs, whereas a scanner with
eight cores would run up to 1024 simultaneous jobs to speed up processing.
A node will typically require somewhere between 256 and 512MB of memory,
and thus can safely run on a VM with 2GB memory (or less).


********************
Source Code Location
********************

.. This needs to change once we have released Warble

*Apache Warble does not currently have any releases.*
*You are however welcome to try out the development version.*

For the time being, we recommend that you use the ``master`` branch for
testing Warble. This applies to both scanners and the server.

The Warble Server can be found via our source repository at
https://github.com/apache/incubator-warble-server

The Warble Node Application can be found via:
https://github.com/apache/incubator-warble-node


*********************
Installing the Server
*********************

###############
Pre-requisites
###############

Before you install the Warble Server, please ensure you have the
following components installed and set up:

- A web server of your choice (Apache HTTP Server, NGINX, lighttp etc)
- Python 3.4 or newer with the following libraries installed:
- - yaml
- - certifi
- - sqlite3
- - bcrypt
- - cryptography >= 2.0.0
- Gunicorn for Python 3.x (often called gunicorn3) or mod_wsgi

###########################################
Configuring and Priming the Warble Server
###########################################
Once you have the components installed and Warble Server downloaded, you will
need to prime the databases and create a configuration file.

Assuming you wish to install warble in /opt/warble, you would set it
up by issuing the following:

- ``git clone https://github.com/apache/incubator-warble-server.git /opt/warble``
- ``cd /opt/warble/setup``
- ``python3 setup.py``
- Enter the configuration parameters the setup process asks for

This will set up the database, the configuration file, and create your
initial administrator account for the UI. You can later on do additional
configuration of the data server by editing the ``api/yaml/warble.yaml``
file.

#####################
Setting up the Web UI
#####################

Once you have finished the initial setup, you will need to enable the
web UI. Warble is built as a WSGI application, and as such you can
use mod_wsgi for apache, or proxy to Gunicorn. In this example, we will
be using the Apache HTTP Server and proxy to Gunicorn:

- Make sure you have mod_proxy and mod_proxy_http loaded (on
  debian/ubuntu, you would run: `a2enmod proxy_http`)
- Set up a virtual host in Apache:

::

   <VirtualHost *:80>
      # Set this to your domain, or add warble.localhost to /etc/hosts
      ServerName warble.localhost
      DocumentRoot /opt/warble/ui/
      # Proxy to gunicorn for /api/ below:
      ProxyPass /api/ http://localhost:8000/api/
   </VirtualHost>

- Launch gunicorn as a daemon on port 8000:

::

   cd /opt/warble/api
   gunicorn -w 10 -b 127.0.0.1:8000 handler:application -t 120 -D

Once httpd is (re)started, you should be able to browse to your new
Warble instance.


*******************
Installing Nodes
*******************

##############
Pre-requisites
##############


The Warble Nodes rely on the following packages:

- Python >= 3.4 with the following packages:
- - python3-yaml
- - python3-ldap
- - python3-dns

Custom node tests may require additional packages.

###########################
Configuring a node
###########################

First, check out the node source in a file path of your choosing:

``git clone https://github.com/apache/incubator-warble-node.git``

Then edit the ``conf/config.yaml`` file to point towards the 
proper Warble Master server.

Then fire up the node software as a daemon:

``python3 node.py start``

Warble Node apps will, when run the first time, set up an async 
key pair for encryption and verification, and request a spot in
the Warble Master node registry. Spots are verified/approved in the
Warble UI, and once completed, the node will receive an API key 
that corresponds with its ID and key pair, and get to work. 
It is worth noting, that the Warble node software needs write access 
to the configuration directory on disk, so it can store the API key and
async key pair.