www/status/README.md - whimsy - Git at Google

 Monitoring
 ==========

 The [current Whimsy server status](https://whimsy.apache.org/status/) is represented
 as a tree of named nodes, and created by the [status monitoring code](https://github.com/apache/whimsy/tree/master/www/status/).

 Nodes, names, and strings
 -------------------------

 Each major branch is produced by a [monitor](monitors).  Each monitor can
 return a tree of nodes, or a single String, or an array of Strings.  The name
 of the monitor is used as the name of the node produced.

 Leaf nodes consist of a String, an array of Strings, or a Hash where one
 element in the Hash has a key of `data` with a value of either a String or an
 array of Strings.

 Non-leaf nodes consist of a Hash where one element in the Hash has a key of
 `data` with a value that is a Hash of names and child nodes.

 Levels
 ------

 Each node is associated with a status *level*.  Valid levels are `success`,
 `info`, `warning`, `danger`, and `fatal`.  (The first four levels are modelled
 after Bootstrap [alerts](http://getbootstrap.com/components/#alerts)).

 Default level for valid leaf nodes is `success`.  Invalid leaf nodes (e.g., a
 node consisting of a `nil` value) have a level of `danger`.  Only leaf nodes
 that in the form of a Hash can have levels.  Leaf nodes that are not Hashes
 will be normalized into a Hash with a `level` and `data`.

 Default level for non-leaf nodes is the highest level in children nodes (where
 `fatal` > `danger`, `danger` > `warning`, `warning` > `info` and `info` >
 `success`).  Normally monitors will not assign level values for non-leaf
 nodes.

 Titles
 ------

 Non-leaf nodes have a *title* describing the contents of the children.  Titles
 show up as tooltips in the browser.

 Default for title is either a list or a count of the names of child nodes with
 the highest status.  Again, normally monitors will not assign title values for
 nodes.

 Text
 ----

 Somewhat rare, but a node may have *text* which is used in place of the name
 of the node for display purposes (the name continues to be used to produce the
 anchor id for the element for linking purposes).

 Internally, exceptions returned by a monitor are converted to a leaf node with
 a name of `exception`, a title containing the exception, and data consisting
 of a stack traceback.

 Href
 ----

 Leaf nodes may have a *href* which will be used as the target for the link
 used to display the contents of the leaf node (either a single String or an
 array of Strings).

 Mtime
 -----

 Anchors and the top of each major branch emanating from the root have an
 *mtime* value which indicates when that data was last updated.  This is
 described below in the control flow section below.

 Leaf nodes can have a mtime value in place of data.  Such values will be
 converted to local time and displayed as the last update value.  Hovering over
 such items will show the GMT value of the time specified in ISO-8601 format.

 Control Flow
 ============

 Fetching the [status](https://whimsy.apache.org/status/) web page, which
 can be done either by browsers or pings, results in a call to
 [index.cgi](https://github.com/apache/whimsy/blob/master/www/status/index.cgi).
 If it has been more than 60 seconds since the last status update, index.cgi
 will call
 [monitor.rb](https://github.com/apache/whimsy/blob/master/www/status/monitor.rb).
 StatusMonitor.rb will load and then call each of the monitors defined in the
 [monitors](https://github.com/apache/whimsy/tree/master/www/status/monitors)
 subdirectory.

 Monitors are simple class methods.  Monitors can assume that they are called
 no more often than once a minute, and are passed the normalized results of the
 previous call.

 As monitors are called in response to a ping, they are expected to produce
 results in sub-second time in order to avoid the ping timing out.  (Monitors
 are run in separate threads to minimize the total elapsed time).  Monitors
 that perform activities that take a substantial amount of time may elect to do
 so less frequently than once a minute, and can take advantage of the `mtime`
 values to determine when to do so.

 Results are collected into a hash, and that hash is then normalized.
 Normalization resolves default values for items like levels and titles
 recursively.

 The normalized status is written to disk as [status.json](status.json), and
 used as a response to pings that occur less than a minute after the previous
 status.

 Alerts
 ======

 The Apache Software Foundation infrastructure team uses
 [Nodeping](https://nodeping.com/reports/status/70MTNEPXE6) to monitor
 status.  A dozen+ servers around the world check status regularly,
 and will report failure results to the infrastructure
 [Slack](https://the-asf.slack.com/) channel.  _Important:_ The Infrastructure
 team ensures the underlying VM is up; the Whimsy PMC is responsible for
 the server software running inside the VM.

 There are currently two Nodeping checks:
 - Public facing: this checks the status return from the public URL https://whimsy.apache.org/incubator/podlings/by-age; this gives[Public check results](https://nodeping.com/reports/statusevents/check/2018042000290QH9Q-OZZ2KBZC)
 - Full status: this checks the status return from https://whimsy.apache.org/status/; this gives [Status results](https://nodeping.com/reports/statusevents/check/2018042000290QH9Q-UMFGNACX)

 While the full status for whimsy is represented as a tree of nodes, each
 assigned one of our levels, and containing either child nodes or one or more
 strings, all the infrastructure team is currently concerned with is a boolean
 status (`success` and `info` are treated as success, and `warning` and
 `danger` are treated as failure) and the computed title for the root node.
	Monitoring
	==========

	The [current Whimsy server status](https://whimsy.apache.org/status/) is represented
	as a tree of named nodes, and created by the [status monitoring code](https://github.com/apache/whimsy/tree/master/www/status/).

	Nodes, names, and strings
	-------------------------

	Each major branch is produced by a [monitor](monitors). Each monitor can
	return a tree of nodes, or a single String, or an array of Strings. The name
	of the monitor is used as the name of the node produced.

	Leaf nodes consist of a String, an array of Strings, or a Hash where one
	element in the Hash has a key of `data` with a value of either a String or an
	array of Strings.

	Non-leaf nodes consist of a Hash where one element in the Hash has a key of
	`data` with a value that is a Hash of names and child nodes.

	Levels
	------

	Each node is associated with a status level. Valid levels are `success`,
	`info`, `warning`, `danger`, and `fatal`. (The first four levels are modelled
	after Bootstrap [alerts](http://getbootstrap.com/components/#alerts)).

	Default level for valid leaf nodes is `success`. Invalid leaf nodes (e.g., a
	node consisting of a `nil` value) have a level of `danger`. Only leaf nodes
	that in the form of a Hash can have levels. Leaf nodes that are not Hashes
	will be normalized into a Hash with a `level` and `data`.

	Default level for non-leaf nodes is the highest level in children nodes (where
	`fatal` > `danger`, `danger` > `warning`, `warning` > `info` and `info` >
	`success`). Normally monitors will not assign level values for non-leaf
	nodes.

	Titles
	------

	Non-leaf nodes have a title describing the contents of the children. Titles
	show up as tooltips in the browser.

	Default for title is either a list or a count of the names of child nodes with
	the highest status. Again, normally monitors will not assign title values for
	nodes.

	Text
	----

	Somewhat rare, but a node may have text which is used in place of the name
	of the node for display purposes (the name continues to be used to produce the
	anchor id for the element for linking purposes).

	Internally, exceptions returned by a monitor are converted to a leaf node with
	a name of `exception`, a title containing the exception, and data consisting
	of a stack traceback.

	Href
	----

	Leaf nodes may have a href which will be used as the target for the link
	used to display the contents of the leaf node (either a single String or an
	array of Strings).

	Mtime
	-----

	Anchors and the top of each major branch emanating from the root have an
	mtime value which indicates when that data was last updated. This is
	described below in the control flow section below.

	Leaf nodes can have a mtime value in place of data. Such values will be
	converted to local time and displayed as the last update value. Hovering over
	such items will show the GMT value of the time specified in ISO-8601 format.

	Control Flow
	============

	Fetching the [status](https://whimsy.apache.org/status/) web page, which
	can be done either by browsers or pings, results in a call to
	[index.cgi](https://github.com/apache/whimsy/blob/master/www/status/index.cgi).
	If it has been more than 60 seconds since the last status update, index.cgi
	will call
	[monitor.rb](https://github.com/apache/whimsy/blob/master/www/status/monitor.rb).
	StatusMonitor.rb will load and then call each of the monitors defined in the
	[monitors](https://github.com/apache/whimsy/tree/master/www/status/monitors)
	subdirectory.

	Monitors are simple class methods. Monitors can assume that they are called
	no more often than once a minute, and are passed the normalized results of the
	previous call.

	As monitors are called in response to a ping, they are expected to produce
	results in sub-second time in order to avoid the ping timing out. (Monitors
	are run in separate threads to minimize the total elapsed time). Monitors
	that perform activities that take a substantial amount of time may elect to do
	so less frequently than once a minute, and can take advantage of the `mtime`
	values to determine when to do so.

	Results are collected into a hash, and that hash is then normalized.
	Normalization resolves default values for items like levels and titles
	recursively.

	The normalized status is written to disk as [status.json](status.json), and
	used as a response to pings that occur less than a minute after the previous
	status.

	Alerts
	======

	The Apache Software Foundation infrastructure team uses
	[Nodeping](https://nodeping.com/reports/status/70MTNEPXE6) to monitor
	status. A dozen+ servers around the world check status regularly,
	and will report failure results to the infrastructure
	[Slack](https://the-asf.slack.com/) channel. _Important:_ The Infrastructure
	team ensures the underlying VM is up; the Whimsy PMC is responsible for
	the server software running inside the VM.

	There are currently two Nodeping checks:
	- Public facing: this checks the status return from the public URL https://whimsy.apache.org/incubator/podlings/by-age; this gives[Public check results](https://nodeping.com/reports/statusevents/check/2018042000290QH9Q-OZZ2KBZC)
	- Full status: this checks the status return from https://whimsy.apache.org/status/; this gives [Status results](https://nodeping.com/reports/statusevents/check/2018042000290QH9Q-UMFGNACX)

	While the full status for whimsy is represented as a tree of nodes, each
	assigned one of our levels, and containing either child nodes or one or more
	strings, all the infrastructure team is currently concerned with is a boolean
	status (`success` and `info` are treated as success, and `warning` and
	`danger` are treated as failure) and the computed title for the root node.