feature: instance boot group and readiness (#677)
* feature: instance boot group and readiness
Related https://github.com/apache/cloudstack/pull/13907
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
* address comments and changes
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
* add screens
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
* more comment
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
---------
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
diff --git a/source/_static/images/instance_boot_group_add.png b/source/_static/images/instance_boot_group_add.png
new file mode 100644
index 0000000..bddf6f9
--- /dev/null
+++ b/source/_static/images/instance_boot_group_add.png
Binary files differ
diff --git a/source/_static/images/instance_boot_group_member_add.png b/source/_static/images/instance_boot_group_member_add.png
new file mode 100644
index 0000000..ee7fdf8
--- /dev/null
+++ b/source/_static/images/instance_boot_group_member_add.png
Binary files differ
diff --git a/source/_static/images/instance_boot_group_member_readiness_rules.png b/source/_static/images/instance_boot_group_member_readiness_rules.png
new file mode 100644
index 0000000..93a4650
--- /dev/null
+++ b/source/_static/images/instance_boot_group_member_readiness_rules.png
Binary files differ
diff --git a/source/_static/images/instance_boot_group_members.png b/source/_static/images/instance_boot_group_members.png
new file mode 100644
index 0000000..e95cabd
--- /dev/null
+++ b/source/_static/images/instance_boot_group_members.png
Binary files differ
diff --git a/source/_static/images/instance_boot_groups.png b/source/_static/images/instance_boot_groups.png
new file mode 100644
index 0000000..ea96645
--- /dev/null
+++ b/source/_static/images/instance_boot_groups.png
Binary files differ
diff --git a/source/adminguide/virtual_machines.rst b/source/adminguide/virtual_machines.rst
index 89d0ffe..1b47d1b 100644
--- a/source/adminguide/virtual_machines.rst
+++ b/source/adminguide/virtual_machines.rst
@@ -1235,6 +1235,12 @@
|vm-settings-virtual-tpm-enabled-vmware.png|
Enable or disable vTPM for VMware.
+Instance Boot Groups
+=====================
+
+.. include:: virtual_machines/instance_boot_groups.rst
+
+
Instance Snapshots
==================
diff --git a/source/adminguide/virtual_machines/instance_boot_groups.rst b/source/adminguide/virtual_machines/instance_boot_groups.rst
new file mode 100644
index 0000000..276024c
--- /dev/null
+++ b/source/adminguide/virtual_machines/instance_boot_groups.rst
@@ -0,0 +1,362 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information#
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+An **Instance Boot Group** lets a User or Administrator group a set of Instances
+and Instance Groups together so that they can be started, stopped, and
+rebooted as a single, ordered unit. This is useful for multi-tier
+applications where some Instances depend on others being up and reachable
+before they should boot, for example a database tier that must be running
+before an application tier starts, which in turn must be running before a
+web tier starts.
+
+An Instance Boot Group is owned by an account, just like an Instance, and is
+managed from **Compute > Instance Boot Groups** in the CloudStack UI.
+
+ |instance_boot_groups.png|
+
+.. note:: An Instance Boot Group is a different concept from an Instance
+ Group (**Compute > Instance Groups**), which is a simple, unordered label
+ used to organize Instances. An Instance Group can itself be added as a
+ member of an Instance Boot Group.
+
+Members and Boot Order
+~~~~~~~~~~~~~~~~~~~~~~~
+
+A **member** of an Instance Boot Group is either a single Instance or an
+existing Instance Group. Each member is assigned a **Boot Order**, a
+non-negative integer.
+
+- All members that share the same Boot Order value form a **tier**, and boot
+ (or shut down) together, concurrently.
+- Tiers are processed strictly in order: when starting a group, tier 0 is
+ processed before tier 1, tier 1 before tier 2, and so on. When stopping a
+ group, tiers are processed in the reverse order.
+- An Instance or Instance Group can be a member of only one Instance Boot
+ Group at a time.
+- An Instance that is part of a VNF appliance, or that already belongs to an
+ AutoScale VM group, cannot be added to an Instance Boot Group.
+- The maximum number of members allowed in a single Instance Boot Group is
+ controlled by the domain-scoped configuration
+ ``instance.boot.group.max.members`` (default: 10).
+
+Readiness Rules
+~~~~~~~~~~~~~~~~
+
+By default, CloudStack does not define any readiness rules for the members of
+an Instance Boot Group, so it considers a member of the group "ready"
+as soon as its Instance (or, for an Instance Group member, all of its
+Instances) reaches the Running state. For many applications this is not
+enough. For example, a database Instance may be Running but not yet
+accepting connections.
+
+A **Readiness Rule** lets you define what "ready" really means for a member,
+beyond the Instance's power state. When an Instance Boot Group is started,
+CloudStack waits for a tier's readiness rules to report success before
+moving on to start the next tier.
+
+Readiness Rules can be attached directly to an Instance member or to an
+Instance Group member. A rule attached to an Instance Group is automatically
+**inherited** by every Instance currently in that group, and is evaluated
+individually against each of them.
+
+The following Readiness Rule types are supported:
+
+.. cssclass:: table-striped table-bordered table-hover
+
+======================= ============================= ==========================================================================================
+Rule Type Applies to Description
+======================= ============================= ==========================================================================================
+Ping Instance, Instance Group Checks that the Instance responds to an ICMP ping, issued from the Virtual Router on the
+ Instance's default network. Requires a running Virtual Router on that network.
+PortCheck Instance, Instance Group Checks that a TCP port on the Instance can be connected to, from the Virtual Router on the
+ Instance's default network. Requires the ``port`` detail (1-65535) and, optionally, the
+ ``protocol`` detail (only ``tcp`` is supported).
+GuestAgentLiveness Instance, Instance Group Checks that the QEMU guest agent inside the Instance responds to a liveness ping.
+ Supported on **KVM only**. Requires installing the ``qemu-guest-agent`` package on the
+ guest Instances.
+MemberQuorum Instance Group only Considers the group ready once a threshold of its member Instances are ready, instead of
+ requiring all of them. Requires the ``thresholdtype`` detail (``COUNT`` or ``PERCENTAGE``)
+ and the ``thresholdvalue`` detail.
+======================= ============================= ==========================================================================================
+
+.. note:: ``Ping``, ``GuestAgentLiveness``, and ``MemberQuorum`` are singleton
+ rules, i.e., only one of each can be attached to a given member. Multiple
+ ``PortCheck`` rules can be attached to the same member, one per port.
+
+.. note:: ``Ping`` and ``PortCheck`` rules require the target Instance to
+ have a default network with a Virtual Router (they cannot be used on an
+ L2 network).
+
+If a member has no Readiness Rules of its own and inherits none from an
+owning Instance Group, its readiness is simply its Instance's power state
+(Running or not).
+
+Configuring Instance Boot Group Behaviour
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The following global configuration parameters control how readiness is
+checked while an Instance Boot Group is starting. All of them, except the
+poll interval and the concurrency setting, can be overridden per Instance
+Boot Group when it is created or updated.
+
+.. cssclass:: table-striped table-bordered table-hover
+
+========================================================= ==========================================================================================
+Configuration Description
+========================================================= ==========================================================================================
+instance.boot.group.readiness.timeout.seconds Timeout, in seconds, for a single readiness check attempt. **Default: 300**
+ (overridable per Instance Boot Group)
+instance.boot.group.readiness.max.retry.attempts Maximum number of readiness check attempts for a member before the tier, and the
+ Instance Boot Group start, is considered to have failed. **Default: 5**
+ (overridable per Instance Boot Group)
+instance.boot.group.readiness.initial.delay.seconds Delay, in seconds, after an Instance is started before its first readiness check
+ attempt is made. **Default: 30** (overridable per Instance Boot Group)
+instance.boot.group.readiness.reboot.on.retry Whether a member Instance should be rebooted before each new readiness check retry,
+ instead of simply being re-checked. **Default: false** (overridable per Instance Boot
+ Group)
+instance.boot.group.readiness.poll.interval.seconds Interval, in seconds, between readiness check attempts. **Default: 10** (global only)
+instance.boot.group.readiness.check.concurrency Maximum number of members within a tier whose readiness is checked concurrently.
+ **Default: 10** (global only)
+instance.boot.group.max.members Maximum number of members allowed in a single Instance Boot Group. This is a
+ domain-scoped setting. **Default: 10**
+========================================================= ==========================================================================================
+
+Creating an Instance Boot Group
+---------------------------------
+
+#. Log in to the CloudStack UI.
+#. In the left navigation bar, click **Compute**, then **Instance Boot Groups**.
+#. Click **Add Instance Boot Group**.
+#. Provide a name and, optionally, a description.
+#. Optionally, expand the readiness settings to override the default
+ readiness timeout, maximum retry attempts, initial delay, or
+ reboot-on-retry behaviour for this Instance Boot Group only.
+#. Click **OK**.
+
+ |instance_boot_group_add.png|
+
+Using the API:
+
+.. code:: bash
+
+ cmk create instancebootgroup name=web-app-stack description="DB, App and Web tiers"
+
+Adding and Managing Members
+------------------------------
+
+Once an Instance Boot Group has been created, Instances or Instance Groups
+can be added to it as members, each with its own Boot Order.
+
+ |instance_boot_group_members.png|
+
+Adding a Member
+~~~~~~~~~~~~~~~~
+
+#. Open the Instance Boot Group and go to the **Members** tab.
+#. Click **Add Member**.
+#. Choose whether the member is an Instance or an Instance Group, then
+ select it from the list.
+#. Enter the Boot Order for the new member.
+#. Click **OK**.
+
+ |instance_boot_group_member_add.png|
+
+Using the API:
+
+.. code:: bash
+
+ # Add an Instance as a member of tier 0 (starts first)
+ cmk add membertoinstancebootgroup id=<boot-group-id> virtualmachineid=<vm-id> order=0
+
+ # Add an Instance Group as a member of tier 1 (starts after tier 0)
+ cmk add membertoinstancebootgroup id=<boot-group-id> instancegroupid=<instance-group-id> order=1
+
+Changing a Member's Boot Order
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+From the **Members** tab, use the **Boot Order** action on a member row to
+move it to a different tier. Any other members between the old and new
+position are automatically shifted to keep the ordering consistent.
+
+Using the API:
+
+.. code:: bash
+
+ cmk update instancebootgroupmember id=<member-id> order=2
+
+Removing a Member
+~~~~~~~~~~~~~~~~~~
+
+Removing a member from an Instance Boot Group only removes it from the
+group (and deletes any Readiness Rules attached to it); the Instance or
+Instance Group itself, and its running state, are not affected.
+
+.. code:: bash
+
+ cmk remove instancebootgroupmember id=<member-id>
+
+Listing Members and Their Status
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The **Members** tab lists all members of an Instance Boot Group sorted by
+Boot Order, along with a readiness status badge (Ready, NotReady, Error, or
+Unknown) for each. An Instance Group member row can be expanded to see the
+readiness of each Instance inside it.
+
+.. code:: bash
+
+ cmk list instancebootgroupmembers bootgroupid=<boot-group-id> details=readiness,children
+
+.. note:: Computing readiness details is not free, so it is only returned
+ when explicitly requested (``details=readiness``). Add
+ ``ignoreinstancestate=true`` to see the last cached rule result even
+ while the Instance is stopped; by default, a stopped Instance is always
+ reported NotReady regardless of the cached result.
+
+Configuring Readiness Rules for a Member
+-------------------------------------------
+
+#. From the **Members** tab, use the **Manage Readiness Rules** action on
+ the member you want to configure.
+#. Click **Add Rule**, choose a Rule Type, and fill in the fields relevant
+ to that type (for example, the port and protocol for a ``PortCheck``
+ rule, or the threshold type and value for a ``MemberQuorum`` rule on an
+ Instance Group).
+#. Click **OK**.
+
+Rules inherited from an owning Instance Group are shown read-only, tagged
+**Inherited**; they must be removed from the Instance Group member instead.
+
+ |instance_boot_group_member_readiness_rules.png|
+
+.. note:: When adding a ``GuestAgentLiveness`` rule, make sure the QEMU
+ guest agent(``qemu-guest-agent`` package) is installed, running, and
+ responsive inside the guest Instance, the rule can only report Ready
+ once the agent answers.
+
+Using the API:
+
+.. code:: bash
+
+ # A TCP port check on an Instance
+ cmk create instancebootgroupreadinessrule bootgroupid=<boot-group-id> \
+ virtualmachineid=<vm-id> ruletype=PortCheck \
+ details[0].port=1433 details[0].protocol=tcp
+
+ # A member quorum rule on an Instance Group (ready once 75% of its Instances are ready)
+ cmk create instancebootgroupreadinessrule bootgroupid=<boot-group-id> \
+ instancegroupid=<instance-group-id> ruletype=MemberQuorum \
+ details[0].thresholdtype=PERCENTAGE details[0].thresholdvalue=75
+
+ cmk list instancebootgroupreadinessrules bootgroupid=<boot-group-id>
+
+ cmk update instancebootgroupreadinessrule id=<rule-id> enabled=false
+
+ cmk delete instancebootgroupreadinessrule id=<rule-id>
+
+.. note:: The rule type of an existing Readiness Rule cannot be changed;
+ only its name, enabled state, and details can be updated. Delete and
+ re-create the rule to change its type.
+
+Starting, Stopping, and Rebooting an Instance Boot Group
+-------------------------------------------------------------
+
+Instance Boot Groups are started, stopped, and rebooted as a whole from the
+**Instance Boot Groups** list or detail view, or via the API. These are
+asynchronous operations.
+
+- **Start**: tiers are started in ascending Boot Order. All members of a
+ tier are started concurrently, and CloudStack waits for the tier to
+ become ready based on its members' Readiness Rules before starting
+ the next tier. If a member does not become ready within the configured
+ number of retry attempts, or fails to start, the whole start operation is
+ halted. Instances that were already started before the failure are left
+ running; they are not automatically stopped again.
+- **Stop**: tiers are stopped in descending Boot Order, the reverse of
+ start. Readiness Rules are not checked while stopping, and a failure to
+ stop one member does not prevent CloudStack from continuing to stop the
+ remaining tiers.
+- **Reboot**: equivalent to a stop followed by a start, i.e. the whole
+ group is stopped in descending order and then started again in ascending,
+ readiness-gated order.
+
+Using the API:
+
+.. code:: bash
+
+ cmk start instancebootgroup id=<boot-group-id>
+
+ cmk stop instancebootgroup id=<boot-group-id> forced=true
+
+ cmk reboot instancebootgroup id=<boot-group-id>
+
+Deleting an Instance Boot Group
+------------------------------------
+
+Deleting an Instance Boot Group removes its members and any Readiness
+Rules attached to them; the Instances and Instance Groups themselves are
+left untouched and continue running or stopped as they were.
+
+.. code:: bash
+
+ cmk delete instancebootgroup id=<boot-group-id>
+
+Instance Boot Group Events
+---------------------------------
+
+The following events are logged for auditing and monitoring Instance Boot
+Group activity:
+
+.. cssclass:: table-striped table-bordered table-hover
+
+================================================= ======================================================
+Event Type Description
+================================================= ======================================================
+INSTANCE.BOOT.GROUP.CREATE An Instance Boot Group was created
+INSTANCE.BOOT.GROUP.UPDATE An Instance Boot Group was updated
+INSTANCE.BOOT.GROUP.DELETE An Instance Boot Group was deleted
+INSTANCE.BOOT.GROUP.START An Instance Boot Group was started
+INSTANCE.BOOT.GROUP.STOP An Instance Boot Group was stopped
+INSTANCE.BOOT.GROUP.REBOOT An Instance Boot Group was rebooted
+INSTANCE.BOOT.GROUP.MEMBER.ADD A member was added to an Instance Boot Group
+INSTANCE.BOOT.GROUP.MEMBER.REMOVE A member was removed from an Instance Boot Group
+INSTANCE.BOOT.GROUP.READINESS.RULE.CREATE A Readiness Rule was created
+INSTANCE.BOOT.GROUP.READINESS.RULE.UPDATE A Readiness Rule was updated
+INSTANCE.BOOT.GROUP.READINESS.RULE.DELETE A Readiness Rule was deleted
+================================================= ======================================================
+
+Limitations
+-----------
+
+#. An Instance or Instance Group can belong to at most one Instance Boot
+ Group at a time.
+#. Instances that are part of a VNF appliance, or already belong to an
+ AutoScale VM group, cannot be added to an Instance Boot Group.
+#. The ``GuestAgentLiveness`` Readiness Rule requires the QEMU guest agent
+ and is supported on KVM Instances only. It requires installing the
+ ``qemu-guest-agent`` package on the guest Instances.
+#. ``Ping`` and ``PortCheck`` Readiness Rules require a running Virtual
+ Router on the Instance's default network, and cannot be used on Instances
+ connected only to an L2 network.
+#. If starting an Instance Boot Group is halted because a tier fails to
+ become ready, any Instances already started in earlier tiers are left
+ running. The operation is not automatically rolled back.
+
+
+.. |instance_boot_groups.png| image:: /_static/images/instance_boot_groups.png
+.. |instance_boot_group_add.png| image:: /_static/images/instance_boot_group_add.png
+.. |instance_boot_group_members.png| image:: /_static/images/instance_boot_group_members.png
+.. |instance_boot_group_member_add.png| image:: /_static/images/instance_boot_group_member_add.png
+.. |instance_boot_group_member_readiness_rules.png| image:: /_static/images/instance_boot_group_member_readiness_rules.png
