.. Licensed to the Apache Software Foundation (ASF) under one
   or more contributor license agreements.  See the NOTICE file
   distributed with this work for additional information#
   regarding copyright ownership.  The ASF licenses this file
   to you under the Apache License, Version 2.0 (the
   "License"); you may not use this file except in compliance
   with the License.  You may obtain a copy of the License at
   http://www.apache.org/licenses/LICENSE-2.0
   Unless required by applicable law or agreed to in writing,
   software distributed under the License is distributed on an
   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
   KIND, either express or implied.  See the License for the
   specific language governing permissions and limitations
   under the License.


Storage Setup
=============

CloudStack is designed to work with a wide variety of commodity and
enterprise-grade storage. Local disk may be used as well, if supported
by the selected hypervisor. Storage type support for guest virtual disks
differs based on hypervisor selection.

=============  ==============================  ==================  ===================================
Storage Type   XenServer                       vSphere             KVM
=============  ==============================  ==================  ===================================
NFS            Supported                       Supported           Supported
iSCSI          Supported                       Supported via VMFS  Supported via Clustered Filesystems
Fiber Channel  Supported via Pre-existing SR   Supported           Supported via Clustered Filesystems
Local Disk     Supported                       Supported           Supported
=============  ==============================  ==================  ===================================

The use of the Cluster Logical Volume Manager (CLVM) for KVM is not
officially supported with CloudStack.


Small-Scale Setup
-----------------

In a small-scale setup, a single NFS server can function as both primary
and secondary storage. The NFS server just needs to export two separate
shares, one for primary storage and the other for secondary storage.


Secondary Storage
-----------------

CloudStack is designed to work with any scalable secondary storage
system. The only requirement is the secondary storage system supports
the NFS protocol.

.. note::
   The storage server should be a machine with a large number of disks. The 
   disks should ideally be managed by a hardware RAID controller. Modern 
   hardware RAID controllers support hot plug functionality independent of the 
   operating system so you can replace faulty disks without impacting the 
   running operating system.


Example Configurations
----------------------

In this section we go through a few examples of how to set up storage to
work properly on a few types of NFS and iSCSI storage systems.


Linux NFS on Local Disks and DAS
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This section describes how to configure an NFS export on a standard
Linux installation. The exact commands might vary depending on the
operating system version.

#. Install the RHEL/CentOS distribution on the storage server.

#. If the root volume is more than 2 TB in size, create a smaller boot
   volume to install RHEL/CentOS. A root volume of 20 GB should be
   sufficient.

#. After the system is installed, create a directory called /export.
   This can each be a directory in the root partition itself or a mount
   point for a large disk volume.

#. If you have more than 16TB of storage on one host, create multiple
   EXT3 file systems and multiple NFS exports. Individual EXT3 file
   systems cannot exceed 16TB.

#. After /export directory is created, run the following command to
   configure it as an NFS export.

   .. sourcecode:: bash

      # echo "/export <CIDR>(rw,async,no_root_squash,no_subtree_check)" >> /etc/exports

   Adjust the above command to suit your deployment needs.

   -  **Limiting NFS export.** It is highly recommended that you limit
      the NFS export to a particular subnet by specifying a subnet mask
      (e.g.,”192.168.1.0/24”). By allowing access from only within the
      expected cluster, you avoid having non-pool member mount the
      storage. The limit you place must include the management
      network(s) and the storage network(s). If the two are the same
      network then one CIDR is sufficient. If you have a separate
      storage network you must provide separate CIDR’s for both or one
      CIDR that is broad enough to span both.

      The following is an example with separate CIDRs:

      .. sourcecode:: bash

         /export 192.168.1.0/24(rw,async,no_root_squash,no_subtree_check) 10.50.1.0/24(rw,async,no_root_squash,no_subtree_check)

   -  **Removing the async flag.** The async flag improves performance
      by allowing the NFS server to respond before writes are committed
      to the disk. Remove the async flag in your mission critical
      production deployment.

#. Run the following command to enable NFS service.

   .. sourcecode:: bash

      # chkconfig nfs on

#. Edit the /etc/sysconfig/nfs file and uncomment the following lines.

   .. sourcecode:: bash

      LOCKD_TCPPORT=32803
      LOCKD_UDPPORT=32769
      MOUNTD_PORT=892
      RQUOTAD_PORT=875
      STATD_PORT=662
      STATD_OUTGOING_PORT=2020

#. Edit the /etc/sysconfig/iptables file and add the following lines at
   the beginning of the INPUT chain.

   .. sourcecode:: bash

      -A INPUT -m state --state NEW -p udp --dport 111 -j ACCEPT
      -A INPUT -m state --state NEW -p tcp --dport 111 -j ACCEPT
      -A INPUT -m state --state NEW -p tcp --dport 2049 -j ACCEPT
      -A INPUT -m state --state NEW -p tcp --dport 32803 -j ACCEPT
      -A INPUT -m state --state NEW -p udp --dport 32769 -j ACCEPT
      -A INPUT -m state --state NEW -p tcp --dport 892 -j ACCEPT
      -A INPUT -m state --state NEW -p udp --dport 892 -j ACCEPT
      -A INPUT -m state --state NEW -p tcp --dport 875 -j ACCEPT
      -A INPUT -m state --state NEW -p udp --dport 875 -j ACCEPT
      -A INPUT -m state --state NEW -p tcp --dport 662 -j ACCEPT
      -A INPUT -m state --state NEW -p udp --dport 662 -j ACCEPT

#. Reboot the server.

   An NFS share called /export is now set up.

.. note::
   When copying and pasting a command, be sure the command has pasted as a 
   single line before executing. Some document viewers may introduce unwanted 
   line breaks in copied text.


Linux NFS on iSCSI
~~~~~~~~~~~~~~~~~~

Use the following steps to set up a Linux NFS server export on an iSCSI
volume. These steps apply to RHEL/CentOS 5 distributions.

#. Install iscsiadm.

   .. sourcecode:: bash

      # yum install iscsi-initiator-utils
      # service iscsi start
      # chkconfig --add iscsi
      # chkconfig iscsi on

#. Discover the iSCSI target.

   .. sourcecode:: bash

      # iscsiadm -m discovery -t st -p <iSCSI Server IP address>:3260

   For example:

   .. sourcecode:: bash

      # iscsiadm -m discovery -t st -p 172.23.10.240:3260 172.23.10.240:3260,1 iqn.2001-05.com.equallogic:0-8a0906-83bcb3401-16e0002fd0a46f3d-rhel5-test

#. Log in.

   .. sourcecode:: bash

      # iscsiadm -m node -T <Complete Target Name> -l -p <Group IP>:3260

   For example:

   .. sourcecode:: bash

      # iscsiadm -m node -l -T iqn.2001-05.com.equallogic:83bcb3401-16e0002fd0a46f3d-rhel5-test -p 172.23.10.240:3260

#. Discover the SCSI disk. For example:

   .. sourcecode:: bash

      # iscsiadm -m session -P3 | grep Attached
      Attached scsi disk sdb State: running

#. Format the disk as ext3 and mount the volume.

   .. sourcecode:: bash

      # mkfs.ext3 /dev/sdb
      # mkdir -p /export
      # mount /dev/sdb /export

#. Add the disk to /etc/fstab to make sure it gets mounted on boot.

   .. sourcecode:: bash

      /dev/sdb /export ext3 _netdev 0 0

Now you can set up /export as an NFS share.

-  **Limiting NFS export.** In order to avoid data loss, it is highly
   recommended that you limit the NFS export to a particular subnet by
   specifying a subnet mask (e.g.,”192.168.1.0/24”). By allowing access
   from only within the expected cluster, you avoid having non-pool
   member mount the storage and inadvertently delete all its data. The
   limit you place must include the management network(s) and the
   storage network(s). If the two are the same network then one CIDR is
   sufficient. If you have a separate storage network you must provide
   separate CIDRs for both or one CIDR that is broad enough to span
   both.

   The following is an example with separate CIDRs:

   .. sourcecode:: bash

      /export 192.168.1.0/24(rw,async,no_root_squash,no_subtree_check) 10.50.1.0/24(rw,async,no_root_squash,no_subtree_check)

-  **Removing the async flag.** The async flag improves performance by
   allowing the NFS server to respond before writes are committed to the
   disk. Remove the async flag in your mission critical production
   deployment.