Upgrading linearstore from pre-partition to post-partition versions
===================================================================

Introduction
============

Partitions were added to linearstore in QPID-5671, and linearstore was
made the default store in QPID-6394. This change resulted in a changed
directory structure. Any user wishing to preserve data through this update
needs to update their directory structure.

Initially, this was done automatically by the store, but the inability
to roll back the change made it risky, and this was removed (QPID-6303)
in favor of using mutually exclusive directory structures for the pre-
and post-partition stores. This way, original data is not affected by
running the new store, and similarly, if the older pre-partition store
is run, it will continue using the old directory structure. However, if
queues and data exist which must be migrated from the old to the new
structure, then it must be done by hand as described below.

Partitions
==========

Linearstore introduces the concept of store partitions. These allow different
physical media (eg rotating magnetic media and solid state drives) to be
mapped to different locations within the linearstore directory structure as
partitions. Persistent queues may then be created and which can be directed
to use a particular type of storage through directing which partition it is
using for persistence. So it is possible, therefore, to have one queue using
magnetic media (where it does not have high performance/low latency
requirements, while another uses (for example) more expensive and limited
capacity solid state drives.

Each partition exists as a directory within the qls top-level store directory
labeled "pNNN" where NNN is a zero-padded integer starting at
001. p001 is the default partition and is on the same disk that the
system is installed on. By mounting other drives/media to "qls/p002",
"qls/p003", etc, other media may by used by some queues and not others.
Each of these partition directories then contain the empty file pool
used by the store.

Changes to the store directory structure
========================================

Pre-partition
-------------

qls
  ├─ dat (contains Berkeley DB database files)
  ├─ jrnl
  │    ├── <queue_name_1> (contains in-use journal files belonging to queue_name_1)
  │    ├── <queue_name_2> (contains in-use journal files belonging to queue_name_2)
  │    ├── <queue_name_3> (contains in-use journal files belonging to queue_name_3)
  │    ...
  ├─ tpl (contains in-use journal files belonging to the TPL)
  └─ p001
       └─ efp
            └── 2028k (contains empty/returned journal files)

Notes:
1. Although directory p001 exists, it is only a partition placeholder
   on the system disk (ie wherever the store directory qls is located)
   and is the only partition used. No other partition directory will be
   recognized or used.
2. The "2048k" Empty File Pool (EFP) is the default 2048KiB file size
   created when a new store is created. Other sizes may be set as the
   default using the --efp-file-size parameter. Allowed sizes must be a
   multiple of 4KiB (the minimum allowed journal file size).
3. As journal files are needed for a queue, they are moved into the queue
   directory ("qls/jrnl/<queue_name>"), and when they are no longer needed
   (ie, all messages contained within the file are dequeued), then it is
   returned to the correct efp directory.

Post-partition
--------------

qls
  ├─ dat2 (contains Berkeley DB database files)
  ├─ jrnl2
  │    ├─ <queue_name_1> (contains links to in-use journal files belonging to queue_name_1)
  │    ├─ <queue_name_2> (contains links to in-use journal files belonging to queue_name_2)
  │    ├─ <queue_name_3> (contains links to in-use journal files belonging to queue_name_3)
  │    ...
  ├─ tpl2 (contains in-use journal files belonging to the TPL)
  ├─ p001
  │    └─ efp
  │         ├─ 2028k (contains empty/returned journal files)
  │         │    ├─ in_use (contains in-use journal files belonging to all queues
  │         │    │          using this partition and size)
  │         │    └─ returned (future use, currently remains empty)
  │         ├─ <size_1>k
  │         │    ├─ in_use
  │         │    └─ returned
  │         ├─ <size_2>k
  │         │    ├─ in_use
  │         │    └─ returned
  │         ...
  ├─ p002
  │    └─ efp
  │         ├─ <size_1>k
  │         │    ├─ in_use
  │         │    └─ returned
  │         ├─ <size_2>k
  │         │    ├─ in_use
  │         │    └─ returned
  │         ...
  ├─ p003
  │    └─ efp
  │         ├─ <size_1>k
  │         │    ├─ in_use
  │         │    └─ returned
  │         ├─ <size_2>k
  │         │    ├─ in_use
  │         │    └─ returned
  │         ...
  ...

Upgrade process
===============

IT IS RECOMMENDED THAT A CLEAN UPDATE BE PERFORMED. This means that all
queues should be purged and deleted, and the broker restarted without
any existing queues.

If you have queues (with or without messages) which MUST be recovered, then
it will be necessary to perform the following manual upgrade procedure. It
may be error-prone if not done carefully, and could be tedious if there are
many queues.

Summary
-------
* Stop broker
* BACK UP THE STORE (in case of disaster)
* Create necessary directory structure
* Copy the DB4 database files to the new location
* For each queue (and the TPL):
  * Copy journal files to default partition in-use dir
  * Create links in new queue dirs to copied journal files
* Upgrade broker
* Restart broker

1. Stop the broker
------------------
How this is done depends on the type and version of Linux the broker is
installed on. For Fedora/RedHat:

If running as a service:
# service qpidd stop

To stop a daemon:
# qpidd -k

or send a TERM signal to the qpidd process.

2. Back up the (old) store
--------------------------
This can be achieved in any number of ways, but if your data/messages
are important and cannot be lost, don't neglect to do this before
attempting any changes to the store directory.

A convenient way to store a backup on a thumb drive might be:
# tar -czf /run/media/<userid>/<thub-drive-mount-name>/qls-<date/time>-backup.tar.gz <abs-path-to-store-dir>/qls

3. Create necessary new directory structure
-------------------------------------------
Create the following directories under the qls directory:

# mkdir <path-to-store-dir>/qls/dat2
# mkdir <path-to-store-dir>/qls/jrnl2
# mkdir <path-to-store-dir>/qls/tpl2

The p001 directory itself remains unchanged, but add the following
subdirectories to each efp/<size>k directory present:

# mkdir <path-to-store-dir>/qls/p001/efp/<size>k/in-use
# mkdir <path-to-store-dir>/qls/p001/efp/<size>k/returned

4. Copy the DB4 database files to the new location
--------------------------------------------------
Other than the location of the database, these files have not changed in any
way, so only a simple copy is needed:

# cp <path-to-store-dir>/qls/dat/* <path-to-store-dir>/qls/dat2/

5. Transfer the queues from the old directory structure to the new directory structure
--------------------------------------------------------------------------------------
NOTE: In this step, all journal files will be pooled in the in-use directory
and symbolic links created to them in the jrnl2/<queue> directory. It is recommended
that this procedure is completed one queue at a time so as not to confuse files when
creating symbolic links.

For each queue (<path-to-store-dir>/qls/jrnl/<queue-name>) directory:

a. Make careful note of the file names in the queue directory. They are all
   UUID names, so it is not easy to identify the queue to which they belong
   outside of the directory that contains them.

b. Copy all of the files into the appropriately sized EFP in_use directory.
   By default, all the files should be from the 2014k-sized pool in p001, but
   double-check both the file sizes and the available pools in the old
   directory structure. If this is so:
   
   # cp <path-to-store-dir>/qls/jrnl/<queue-name>/*.jrnl <path-to-store-dir>/qls/p001/2048k/in_use/
   
   Make sure file owner/permissions are not changed.
 
c. Create a new queue and create symbolic links to the journal files just
   copied above:

   # mkdir  <path-to-store-dir>/qls/jrnl/<queue-name>

   For each copied file noted in step a. above:
   
   # ln -s <abs-path-to-store-dir>/qls/p001/2048k/in_use/<uuid_file_name.jrnl> <path-to-store-dir>/qls/jrnl/<queue-name>/
   
6. Transfer the TPL from the old directory structure to the new directory structure
-----------------------------------------------------------------------------------
NOTE: The TPL directory may not be present or empty in some cases where transactions
are not in use. If this is so, this step may be skipped.

If the TPL is present and contains journal files, then it should be transferred to
the new structure. It is a special queue and is identical to any regular queue journal
other than its location in the directory structure:

a. Make careful note of the file names in the qls/tpl directory. They are all
   UUID names, so it is not easy to identify the queue to which they belong
   outside of the directory that contains them.

b. Copy all of the files into the appropriately sized EFP in_use directory.
   By default, all the files should be from the 2014k-sized pool in p001, but
   double-check both the file sizes and the available pools in the old
   directory structure. If this is so:
   
   # cp <path-to-store-dir>/qls/jrnl/tpl/*.jrnl <path-to-store-dir>/qls/p001/2048k/in_use/
   
   Make sure file owner/permissions are not changed.
 
c. Create a new queue and create symbolic links to the journal files just
   copied above:

   For each copied file noted in step a. above:
   
   # ln -s <abs-path-to-store-dir>/qls/p001/2048k/in_use/<uuid_file_name.jrnl> <path-to-store-dir>/qls/tpl2/
   
7. Restore ownership, permissions and SELinux contexts to the new directories
-----------------------------------------------------------------------------
These commands can be run recursively to the entire qls directory:

# chown -R qpidd:qpidd <abs-path-to-store-dir>/qls
# restorecon -FvvR <abs-path-to-store-dir>/qls

8. Upgrade the broker
---------------------
Perform the upgrade to the broker and store that will provide partition
functionality. How this is done depends on the distribution of Linux, and
will usually be achieved via the package manager for that distribution.

9. Restart the broker
---------------------
At this point, the broker should recover all the queues from the new directory structure.
The old (original) directories are no longer in use, and once the upgrade has been
verified as successful, they may be deleted.

NOTE: It might be helpful to use log level info+ when first starting the broker, as
the linearstore uses this level to print a lot of recovery detail which might be helpful
in troubleshooting if it should be necessary. This is done through the use of 
qpidd command-line parameter "--log-enable info+" or an equivalent setting in the
broker configuration file.

10 Troubleshooting
------------------
If the upgrade was not successful, or errors occur upon broker restart, then