Merge branch 'develop'
diff --git a/admin/BackingUpandRestoringHAWQDatabases.md b/admin/BackingUpandRestoringHAWQDatabases.md
index 1087aad..9f6fe0a 100644
--- a/admin/BackingUpandRestoringHAWQDatabases.md
+++ b/admin/BackingUpandRestoringHAWQDatabases.md
@@ -53,19 +53,19 @@
To create a backup archive for database `mydb`:
-```
+```shell
$ pg_dump -Ft -f mydb.tar mydb
```
To create a compressed backup using custom format and compression level 3:
-```
+```shell
$ pg_dump -Fc -Z3 -f mydb.dump mydb
```
To restore from an archive using `pg_restore`:
-```
+```shell
$ pg_restore -d new_db mydb.dump
```
@@ -170,7 +170,7 @@
In this example, issuing the first command creates two folders on two different disks with the same postfix `backup/tpch_20140627`. These folders are labeled as backups of the `tpch` database on 2014-06-27. In the next two commands, the example shows two `gpfdist` instances, one using port 8080, and another using port 8081:
- ```
+ ```shell
sdw1$ mkdir -p /data1/gpadmin/backup/tpch_20140627 /data2/gpadmin/backup/tpch_20140627
sdw1$ gpfdist -d /data1/gpadmin/backup/tpch_20140627 -p 8080 &
sdw1$ gpfdist -d /data2/gpadmin/backup/tpch_20140627 -p 8081 &
@@ -178,7 +178,7 @@
2. Save the schema for the database:
- ```
+ ```shell
master_host$ pg_dump --schema-only -f tpch.schema tpch
master_host$ scp tpch.schema sdw1:/data1/gpadmin/backup/tpch_20140627
```
@@ -187,19 +187,21 @@
3. Create a writable external table for each table in the database:
- ```
+ ```shell
master_host$ psql tpch
+ ```
+ ```sql
tpch=# create writable external table wext_orders (like orders)
- tpch-# location ('gpfdist://sdw1:8080/orders1.csv', 'gpfdist://sdw1:8081/orders2.csv') format 'CSV';
+ tpch-# location('gpfdist://sdw1:8080/orders1.csv', 'gpfdist://sdw1:8081/orders2.csv') format 'CSV';
tpch=# create writable external table wext_lineitem (like lineitem)
- tpch-# location ('gpfdist://sdw1:8080/lineitem1.csv', 'gpfdist://sdw1:8081/lineitem2.csv') format 'CSV';
+ tpch-# location('gpfdist://sdw1:8080/lineitem1.csv', 'gpfdist://sdw1:8081/lineitem2.csv') format 'CSV';
```
The sample shows two tables in the `tpch` database: `orders` and `line item`. The sample shows that two corresponding external tables are created. Specify a location or each `gpfdist` instance in the `LOCATION` clause. This sample uses the CSV text format here, but you can also choose other delimited text formats. For more information, see the `CREATE EXTERNAL TABLE` SQL command.
4. Unload data to the external tables:
- ```
+ ```sql
tpch=# begin;
tpch=# insert into wext_orders select * from orders;
tpch=# insert into wext_lineitem select * from lineitem;
@@ -210,7 +212,7 @@
Find the progress ID and kill the process:
- ```
+ ```shell
sdw1$ ps -ef | grep gpfdist
sdw1$ kill 612368; kill 612369
```
@@ -220,14 +222,14 @@
1. Restart `gpfdist` instances if they aren’t running:
- ```
+ ```shell
sdw1$ gpfdist -d /data1/gpadmin/backup/tpch_20140627 -p 8080 &
sdw1$ gpfdist -d /data2/gpadmin/backup/tpch_20140627 -p 8081 &
```
2. Create a new database and restore the schema:
- ```
+ ```shell
master_host$ createdb tpch2
master_host$ scp sdw1:/data1/gpadmin/backup/tpch_20140627/tpch.schema .
master_host$ psql -f tpch.schema -d tpch2
@@ -235,25 +237,27 @@
3. Create a readable external table for each table:
- ```
+ ```shell
master_host$ psql tpch2
-
- tpch2=# create external table rext_orders (like orders) location ('gpfdist://sdw1:8080/orders1.csv', 'gpfdist://sdw1:8081/orders2.csv') format 'CSV';
- tpch2=# create external table rext_lineitem (like lineitem) location ('gpfdist://sdw1:8080/lineitem1.csv', 'gpfdist://sdw1:8081/lineitem2.csv') format 'CSV';
+ ```
+
+ ```sql
+ tpch2=# create external table rext_orders (like orders) location('gpfdist://sdw1:8080/orders1.csv', 'gpfdist://sdw1:8081/orders2.csv') format 'CSV';
+ tpch2=# create external table rext_lineitem (like lineitem) location('gpfdist://sdw1:8080/lineitem1.csv', 'gpfdist://sdw1:8081/lineitem2.csv') format 'CSV';
```
**Note:** The location clause is the same as the writable external table above.
4. Load data back from external tables:
- ```
+ ```sql
tpch2=# insert into orders select * from rext_orders;
tpch2=# insert into lineitem select * from rext_lineitem;
```
5. Run the `ANALYZE` command after data loading:
- ```
+ ```sql
tpch2=# analyze;
```
@@ -262,7 +266,7 @@
Keep in mind that `gpfdist` is accessed at runtime by the segment instances. Therefore, you must ensure that the HAWQ segment hosts have network access to gpfdist. Since the `gpfdist` program is a web server, to test connectivity you can run the following command from each host in your HAWQ array \(segments and master\):
-```
+```shell
$ wget http://gpfdist_hostname:port/filename
```
@@ -270,37 +274,39 @@
## Using PXF <a id="usingpxf"></a>
-Pivotal Extension Framework \(PXF\) is an extensible framework that allows HAWQ to query external system data. The details of how to install and use PXF can be found in *PXF Installation and Administration*.
+HAWQ Extension Framework \(PXF\) is an extensible framework that allows HAWQ to query external system data. The details of how to install and use PXF can be found in [Working with PXF and External Data](../pxf/PivotalExtensionFrameworkPXF.html).
### Using PXF to Back Up the tpch Database <a id="usingpxftobackupthetpchdatabase"></a>
1. Create a folder on HDFS for this backup:
- ```
+ ```shell
master_host$ hdfs dfs -mkdir -p /backup/tpch-2014-06-27
```
2. Dump the database schema using `pg_dump` and store the schema file in a backup folder:
- ```
+ ```shell
master_host$ pg_dump --schema-only -f tpch.schema tpch
master_host$ hdfs dfs -copyFromLocal tpch.schema /backup/tpch-2014-06-27
```
3. Create a writable external table for each table in the database:
- ```
+ ```shell
master_host$ psql tpch
-
+ ```
+
+ ```sql
tpch=# CREATE WRITABLE EXTERNAL TABLE wext_orders (LIKE orders)
- tpch-# LOCATION ('pxf://namenode_host:51200/backup/tpch-2014-06-27/orders'
+ tpch-# LOCATION('pxf://namenode_host:51200/backup/tpch-2014-06-27/orders'
tpch-# '?Profile=HdfsTextSimple'
tpch-# '&COMPRESSION_CODEC=org.apache.hadoop.io.compress.SnappyCodec'
tpch-# )
tpch-# FORMAT 'TEXT';
tpch=# CREATE WRITABLE EXTERNAL TABLE wext_lineitem (LIKE lineitem)
- tpch-# LOCATION ('pxf://namenode_host:51200/backup/tpch-2014-06-27/lineitem'
+ tpch-# LOCATION('pxf://namenode_host:51200/backup/tpch-2014-06-27/lineitem'
tpch-# '?Profile=HdfsTextSimple'
tpch-# '&COMPRESSION_CODEC=org.apache.hadoop.io.compress.SnappyCodec')
tpch-# FORMAT 'TEXT';
@@ -310,7 +316,7 @@
4. Unload the data to external tables:
- ```
+ ```sql
tpch=# BEGIN;
tpch=# INSERT INTO wext_orders SELECT * FROM orders;
tpch=# INSERT INTO wext_lineitem SELECT * FROM lineitem;
@@ -319,7 +325,7 @@
5. **\(Optional\)** Change the HDFS file replication factor for the backup folder. HDFS replicates each block into three blocks by default for reliability. You can decrease this number for your backup files if you need to:
- ```
+ ```shell
master_host$ hdfs dfs -setrep 2 /backup/tpch-2014-06-27
```
@@ -330,7 +336,7 @@
1. Create a new database and restore the schema:
- ```
+ ```shell
master_host$ createdb tpch2
master_host$ hdfs dfs -copyToLocal /backup/tpch-2014-06-27/tpch.schema .
master_host$ psql -f tpch.schema -d tpch2
@@ -338,13 +344,16 @@
2. Create a readable external table for each table to restore:
- ```
+ ```shell
master_host$ psql tpch2
+ ```
+
+ ```sql
tpch2=# CREATE EXTERNAL TABLE rext_orders (LIKE orders)
- tpch2-# LOCATION ('pxf://namenode_host:51200/backup/tpch-2014-06-27/orders?Profile=HdfsTextSimple')
+ tpch2-# LOCATION('pxf://namenode_host:51200/backup/tpch-2014-06-27/orders?Profile=HdfsTextSimple')
tpch2-# FORMAT 'TEXT';
tpch2=# CREATE EXTERNAL TABLE rext_lineitem (LIKE lineitem)
- tpch2-# LOCATION ('pxf://namenode_host:51200/backup/tpch-2014-06-27/lineitem?Profile=HdfsTextSimple')
+ tpch2-# LOCATION('pxf://namenode_host:51200/backup/tpch-2014-06-27/lineitem?Profile=HdfsTextSimple')
tpch2-# FORMAT 'TEXT';
```
@@ -352,13 +361,13 @@
3. Load data back from external tables:
- ```
+ ```sql
tpch2=# INSERT INTO ORDERS SELECT * FROM rext_orders;
tpch2=# INSERT INTO LINEITEM SELECT * FROM rext_lineitem;
```
4. Run `ANALYZE` after data loading:
- ```
+ ```sql
tpch2=# ANALYZE;
```
diff --git a/admin/ClusterExpansion.md b/admin/ClusterExpansion.md
index 7073ac7..d1df3dd 100644
--- a/admin/ClusterExpansion.md
+++ b/admin/ClusterExpansion.md
@@ -15,7 +15,7 @@
- When you add a new node, install both a DataNode and a physical segment on the new node.
- After adding a new node, you should always rebalance HDFS data to maintain cluster performance.
- Adding or removing a node also necessitates an update to the HDFS metadata cache. This update will happen eventually, but can take some time. To speed the update of the metadata cache, execute **`select gp_metadata_cache_clear();`**.
-- Note that for hash distributed tables, expanding the cluster will not immediately improve performance since hash distributed tables use a fixed number of virtual segments. In order to obtain better performance with hash distributed tables, you must redistribute the table to the updated cluster by either the [ALTER TABLE](/200/hawq/reference/sql/ALTER-TABLE.html) or [CREATE TABLE AS](/200/hawq/reference/sql/CREATE-TABLE-AS.html) command.
+- Note that for hash distributed tables, expanding the cluster will not immediately improve performance since hash distributed tables use a fixed number of virtual segments. In order to obtain better performance with hash distributed tables, you must redistribute the table to the updated cluster by either the [ALTER TABLE](/20/reference/sql/ALTER-TABLE.html) or [CREATE TABLE AS](/20/reference/sql/CREATE-TABLE-AS.html) command.
- If you are using hash tables, consider updating the `default_hash_table_bucket_number` server configuration parameter to a larger value after expanding the cluster but before redistributing the hash tables.
## Adding a New Node to an Existing HAWQ Cluster <a id="task_hawq_expand"></a>
@@ -27,7 +27,7 @@
1. Prepare the target machine by checking operating system configurations and passwordless ssh. HAWQ requires passwordless ssh access to all cluster nodes. To set up passwordless ssh on the new node, perform the following steps:
1. Login to the master HAWQ node as gpadmin. If you are logged in as a different user, switch to the gpadmin user and source the `greenplum_path.sh` file.
- ```
+ ```shell
$ su - gpadmin
$ source /usr/local/hawq/greenplum_path.sh
```
@@ -40,40 +40,40 @@
3. Login to the master HAWQ node as root and source the `greenplum_path.sh` file.
- ```
+ ```shell
$ su - root
$ source /usr/local/hawq/greenplum_path.sh
```
4. Execute the following hawq command to set up passwordless ssh for root on the new host machine:
- ```
+ ```shell
$ hawq ssh-exkeys -e hawq_hosts -x new_hosts
```
5. Create the gpadmin user on the new host\(s\).
- ```
- $ hawq ssh -f new_hosts -e '/usr/sbin/useradd gpadmin’
+ ```shell
+ $ hawq ssh -f new_hosts -e '/usr/sbin/useradd gpadmin'
$ hawq ssh –f new_hosts -e 'echo -e "changeme\changeme" | passwd gpadmin'
```
6. Switch to the gpadmin user and source the `greenplum_path.sh` file again.
- ```
+ ```shell
$ su - gpadmin
$ source /usr/local/hawq/greenplum_path.sh
```
7. Execute the following hawq command a second time to set up passwordless ssh for the gpadmin user:
- ```
+ ```shell
$ hawq ssh-exkeys -e hawq_hosts -x new_hosts
```
8. After setting up passwordless ssh, you can execute the following hawq command to check the target machine's configuration.
- ```
+ ```shell
$ hawq check -f new_hosts
```
@@ -81,7 +81,7 @@
2. Login to the target host machine `sdw4` as the root user. If you are logged in as a different user, switch to the root account:
- ```
+ ```shell
$ su - root
```
@@ -90,9 +90,15 @@
4. Download and install HAWQ on the target machine \(`sdw4`\) as described in the [software build instructions](https://cwiki.apache.org/confluence/display/HAWQ/Build+and+Install) or in the distribution installation documentation.
5. On the HAWQ master node, check current cluster and host information using `psql`.
+ ```shell
+ $ psql -d postgres
```
- #psql -d postgres
- postgres=#select * from gp_segment_configuration;
+
+ ```sql
+ postgres=# select * from gp_segment_configuration;
+ ```
+
+ ```
registration_order | role | status | port | hostname | address
--------------------+------+--------+-------+----------+---------------
-1 | s | u | 5432 | sdw1 | 192.168.2.202
@@ -106,7 +112,7 @@
6. Execute the following command to confirm that HAWQ was installed on the new host:
- ```
+ ```shell
$ hawq ssh -f new_hosts -e "ls -l $GPHOME"
```
@@ -132,7 +138,7 @@
9. Sync the `hawq-site.xml` and `slaves` configuration files to all nodes in the cluster \(as listed in hawq\_hosts\).
- ```
+ ```shell
$ hawq scp -f hawq_hosts hawq-site.xml slaves =:$GPHOME/etc/
```
@@ -144,16 +150,22 @@
12. On `sdw4`, switch to the database user \(for example, `gpadmin`\), and initalize the segment.
- ```
+ ```shell
$ su - gpadmin
$ hawq init segment
```
13. On the master node, check current cluster and host information using `psql` to verify that the new `sdw4` node has initialized successfully.
+ ```shell
+ $ psql -d postgres
```
- #psql -d postgres
+
+ ```sql
postgres=# select * from gp_segment_configuration ;
+ ```
+
+ ```
registration_order | role | status | port | hostname | address
--------------------+------+--------+-------+----------+---------------
-1 | s | u | 5432 | sdw1 | 192.168.2.202
@@ -165,22 +177,31 @@
```
14. To maintain optimal cluster performance, rebalance HDFS data by running the following command:
- <pre><code>$ sudo -u hdfs hdfs balancer -threshold <i>threshhold_value</i>
- </code></pre>
-
+15.
+ ```shell
+ $ sudo -u hdfs hdfs balancer -threshold threshhold_value
+ ```
+
where *threshhold\_value* represents how much a DataNode's disk usage, in percentage, can differ from overall disk usage in the cluster. Adjust the threshold value according to the needs of your production data and disk. The smaller the value, the longer the rebalance time.
>
**Note:** If you do not specify a threshold, then a default value of 20 is used. If the balancer detects that a DataNode is using less than a 20% difference of the cluster's overall disk usage, then data on that node will not be rebalanced. For example, if disk usage across all DataNodes in the cluster is 40% of the cluster's total disk-storage capacity, then the balancer script ensures that a DataNode's disk usage is between 20% and 60% of that DataNode's disk-storage capacity. DataNodes whose disk usage falls within that percentage range will not be rebalanced.
Rebalance time is also affected by network bandwidth. You can adjust network bandwidth used by the balancer by using the following command:
- <pre><code>$ sudo -u hdfs hdfs dfsadmin -setBalancerBandwidth <i>network_bandwith</i></code></pre>
+
+ ```shell
+ $ sudo -u hdfs hdfs dfsadmin -setBalancerBandwidth network_bandwith
+ ```
+
The default value is 1MB/s. Adjust the value according to your network.
15. Speed up the clearing of the metadata cache by using the following command:
+ ```shell
+ $ psql -d postgres
```
- #psql -d postgres
- postgres=#select gp_metadata_cache_clear();
+
+ ```sql
+ postgres=# select gp_metadata_cache_clear();
```
16. After expansion, if the new size of your cluster is greater than or equal \(#nodes >=4\) to 4, change the value of the `output.replace-datanode-on-failure` HDFS parameter in `hdfs-client.xml` to `false`.
@@ -197,7 +218,7 @@
|\> 256 and <= 512|1 \* \#nodes|
|\> 512|512|
-18. If you are using hash distributed tables and wish to take advantage of the performance benefits of using a larger cluster, redistribute the data in all hash-distributed tables by using either the [ALTER TABLE](/200/hawq/reference/sql/ALTER-TABLE.html) or [CREATE TABLE AS](/200/hawq/reference/sql/CREATE-TABLE-AS.html) command. You should redistribute the table data if you modified the `default_hash_table_bucket_number` configuration parameter.
+18. If you are using hash distributed tables and wish to take advantage of the performance benefits of using a larger cluster, redistribute the data in all hash-distributed tables by using either the [ALTER TABLE](/20/reference/sql/ALTER-TABLE.html) or [CREATE TABLE AS](/20/reference/sql/CREATE-TABLE-AS.html) command. You should redistribute the table data if you modified the `default_hash_table_bucket_number` configuration parameter.
**Note:** The redistribution of table data can take a significant amount of time.
diff --git a/admin/ClusterShrink.md b/admin/ClusterShrink.md
index c31702d..154cbe3 100644
--- a/admin/ClusterShrink.md
+++ b/admin/ClusterShrink.md
@@ -19,20 +19,20 @@
1. Login as gpadmin to the node that you wish to remove and source `greenplum_path.sh`.
- ```
+ ```shell
$ su - gpadmin
$ source /usr/local/hawq/greenplum_path.sh
```
2. Make sure that there are no running QEs on the segment. Execute the following command to check for running QE processes:
- ```
- $ ps -ef | grep postres
+ ```shell
+ $ ps -ef | grep postgres
```
In the output, look for processes that contain SQL commands such as INSERT or SELECT. For example:
- ```
+ ```shell
[gpadmin@rhel64-3 ~]$ ps -ef | grep postgres
gpadmin 3000 2999 0 Mar21 ? 00:00:08 postgres: port 40000, logger process
gpadmin 3003 2999 0 Mar21 ? 00:00:03 postgres: port 40000, stats collector process
@@ -44,12 +44,12 @@
3. Stop hawq on this segment by executing the following command:
- ```
+ ```shell
$ hawq stop segment
```
4. On HAWQ master, remove the hostname of the segment from the `slaves` file. Then sync the `slaves` file to all nodes in the cluster by executing the following command:
- ```
+ ```shell
$ hawq scp -f hostfile slaves =: $GPHOME/etc/slaves
```
diff --git a/admin/HAWQFilespacesandHighAvailabilityEnabledHDFS.md b/admin/HAWQFilespacesandHighAvailabilityEnabledHDFS.md
index 48882bd..26ca265 100644
--- a/admin/HAWQFilespacesandHighAvailabilityEnabledHDFS.md
+++ b/admin/HAWQFilespacesandHighAvailabilityEnabledHDFS.md
@@ -30,7 +30,7 @@
1. Use the following SQL query to gather information about the filespace located on HDFS:
- ```
+ ```sql
SELECT
fsname, fsedbid, fselocation
FROM
@@ -78,42 +78,42 @@
1. If you defined a custom port for HAWQ master, export the PGPORT environment variable. For example:
- ```
+ ```shell
export PGPORT=9000
```
1. If you have not configured it already, export the MASTER\_DATA\_DIRECTORY environment variable.
- ```
+ ```bash
export MASTER_DATA_DIRECTORY=/path/to/master/catalog
```
- See [Environment Variables](/200/hawq/reference/HAWQEnvironmentVariables.html) for more information on environment variables.
+ See [Environment Variables](/20/reference/HAWQEnvironmentVariables.html) for more information on environment variables.
1. Disconnect all workload connections. Check the active connection with:
- ```
- psql -p ${PGPORT} -c "select * from pg_catalog.pg_stat_activity" -d template1
+ ```shell
+ $ psql -p ${PGPORT} -c "select * from pg_catalog.pg_stat_activity" -d template1
```
where $\{PGPORT\} corresponds to the port number you optionally customized for HAWQ master.
2. Issue a checkpoint:
- ```
- psql -p ${PGPORT} -c "checkpoint" -d template1
+ ```shell
+ $ psql -p ${PGPORT} -c "checkpoint" -d template1
```
3. Shut down the HAWQ cluster:
- ```
- hawq stop cluster -a -M fast
+ ```shell
+ $ hawq stop cluster -a -M fast
```
4. Copy the master data directory to a backup location:
- ```
- $cp -r ${MASTER_DATA_DIRECTORY} /catalog/backup/location
+ ```shell
+ $ cp -r ${MASTER_DATA_DIRECTORY} /catalog/backup/location
```
The master data directory contains the catalog. Fatal errors can occur due to hardware failure or if you fail to kill a HAWQ process before attempting a filespace location change. Make sure you back this directory up.
@@ -125,13 +125,13 @@
1. If you defined a custom port for HAWQ master, export the PGPORT environment variable. For example:
- ```
+ ```shell
export PGPORT=9000
```
1. Run the following command to move a file space location:
- ```
- hawq filespace --movefilespace default --location=hdfs://hdfs-cluster/hawq_new_filespace
+ ```shell
+ $ hawq filespace --movefilespace default --location=hdfs://hdfs-cluster/hawq_new_filespace
```
If the target original filespace is not the default filespace, replace `default` in command line with the actual filespace name. Replace `hdfs://hdfs-cluster/hawq_new_filespace` with new filespace location.
@@ -152,7 +152,7 @@
1. Edit the ` ${GPHOME}/etc/hdfs-client.xml` file on each segment and add the following NameNode properties:
- ```
+ ```xml
<property>
<name>dfs.ha.namenodes.hdpcluster</name>
<value>nn1,nn2</value>
@@ -192,7 +192,7 @@
2. Change the following parameter in the `$GPHOME/etc/hawq-site.xml` file:
- ```
+ ```xml
<property>
<name>hawq_dfs_url</name>
<value>hdpcluster/hawq_default</value>
@@ -206,7 +206,7 @@
3. Copy the updated configuration files to all nodes in the cluster (as listed in `hawq_hosts`).
- ```
+ ```shell
$ hawq scp -f hawq_hosts hdfs-client.xml hawq-site.xml =:$GPHOME/etc/
```
@@ -214,13 +214,13 @@
1. Restart the HAWQ cluster:
- ```
+ ```shell
$ hawq start cluster -a
```
1. Moving the filespace to a new location renders the standby master catalog invalid. To update the standby, resync the standby master. On the active master, run the following command to ensure that the standby master's catalog is resynced with the active master.
- ```
+ ```shell
$ hawq init standby -n -M fast
```
diff --git a/admin/MasterMirroring.md b/admin/MasterMirroring.md
index 8d90578..c731427 100644
--- a/admin/MasterMirroring.md
+++ b/admin/MasterMirroring.md
@@ -37,7 +37,7 @@
* Data directory created
1. Run the `hawq init standby` utility on the host of the currently active primary master to add a standby master host to your HAWQ system. For example:
- ```
+ ```shell
$ hawq init standby -s new_standby_master_hostname
```
@@ -59,21 +59,21 @@
1. Run the hawq activate standby utility from the standby master host you are activating. For example:
- ```
+ ```shell
$ hawq activate standby
```
1. After you activate the standby master, it becomes the active or primary master for the HAWQ cluster.
1. (Optional, but recommended.) Configure a new standby master. Run the following command:
- ```
+ ```shell
$ hawq init standby -s new_standby_master_hostname
```
where `-s` specifies the hostname of the standby master.
1. Run hawq state to check the status of the cluster:
- ```
+ ```shell
$ hawq state
```
@@ -83,9 +83,7 @@
1. Query the `gp_segment_configuration` table to verify that segments have registered themselves to the new master.
- ```
$ psql dbname -c 'SELECT * FROM gp_segment_configuration;'
- ```
1. Then, check on the status of master mirroring by querying the `gp_master_mirroring` system view. See [Checking on the State of Master Mirroring](#standby_check) for instructions.
@@ -94,9 +92,9 @@
To check on the status of master mirroring, query the `gp_master_mirroring` system view. The view provides information about the walsender process used for HAWQ master mirroring.
-
- $ psql dbname -c 'SELECT * FROM gp_master_mirroring;'
-
+```shell
+$ psql dbname -c 'SELECT * FROM gp_master_mirroring;'
+```
If a standby master has not been set up for the cluster, you will see the following output:
@@ -130,6 +128,8 @@
To resynchronize the standby with the master, execute the following command on the master.
- $ hawq init standby -n
+```shell
+$ hawq init standby -n
+```
This command stops and restarts the master and then synchronizes the standby.
diff --git a/admin/RecommendedMonitoringTasks.md b/admin/RecommendedMonitoringTasks.md
index d5a4fd3..ec67228 100644
--- a/admin/RecommendedMonitoringTasks.md
+++ b/admin/RecommendedMonitoringTasks.md
@@ -20,8 +20,7 @@
<td><p>List segments that are currently down. If any rows are returned, this should generate a warning or alert.</p>
<p>Recommended frequency: run every 5 to 10 minutes</p><p>Severity: IMPORTANT</p></td>
<td>Run the following query in the `postgres` database:
- <pre>code>
-SELECT * FROM gp_segment_configuration
+ <pre><code>SELECT * FROM gp_segment_configuration
WHERE status <> 'u';
</code></pre>
</td>
@@ -189,7 +188,7 @@
</tr>
<tr>
<td>
- <p>Vacuum all system catalogs (tables in the <code>pg_catalog</code> schema) that are approaching <a href="/200/hawq/reference/guc/parameter_definitions.html">vacuum_freeze_min_age</a>.</p>
+ <p>Vacuum all system catalogs (tables in the <code>pg_catalog</code> schema) that are approaching <a href="/20/reference/guc/parameter_definitions.html">vacuum_freeze_min_age</a>.</p>
<p>Recommended frequency: daily</p>
<p>Severity: CRITICAL</p>
</td>
@@ -197,7 +196,7 @@
<p><p>Vacuum an individual table:</p>
<pre><code>VACUUM <<i>table</i>>;</code></pre>
</td>
- <td>After the <a href="/200/hawq/reference/guc/parameter_definitions.html">vacuum_freeze_min_age</a> value is reached, VACUUM will no longer replace transaction IDs with <code>FrozenXID</code> while scanning a table. Perform vacuum on these tables before the limit is reached.</td>
+ <td>After the <a href="/20/reference/guc/parameter_definitions.html">vacuum_freeze_min_age</a> value is reached, VACUUM will no longer replace transaction IDs with <code>FrozenXID</code> while scanning a table. Perform vacuum on these tables before the limit is reached.</td>
</tr>
<td>
<p>Update table statistics.</p>
diff --git a/admin/ambari-admin.md b/admin/ambari-admin.md
index b51d3c4..bee0ca7 100644
--- a/admin/ambari-admin.md
+++ b/admin/ambari-admin.md
Binary files differ
diff --git a/admin/monitor.md b/admin/monitor.md
index 58492c1..af4d138 100644
--- a/admin/monitor.md
+++ b/admin/monitor.md
@@ -25,13 +25,13 @@
The default `hawq state` action is to check segment instances and show a brief status of the valid and failed segments. For example, to see a quick status of your HAWQ system, type:
-```
+```shell
$ hawq state -b
```
You can also display information about the HAWQ master data directory by using `hawq state` with the `-d` option:
-```
+```shell
$ hawq state -d MASTER_DIR
```
@@ -45,31 +45,31 @@
To see the total size of a database \(in bytes\), use the *hawq\_size\_of\_database* view in the *hawq\_toolkit* administrative schema. For example:
-```
+```sql
=> SELECT * FROM hawq_toolkit.hawq_size_of_database
- ORDER BY sodddatname;
+ORDER BY sodddatname;
```
##### Viewing Disk Space Usage for a Table <a id="topic18"></a>
The *hawq\_toolkit* administrative schema contains several views for checking the size of a table. The table sizing views list the table by object ID \(not by name\). To check the size of a table by name, you must look up the relation name \(`relname`\) in the *pg\_class* table. For example:
-```
+```sql
=> SELECT relname AS name, sotdsize AS size, sotdtoastsize
- AS toast, sotdadditionalsize AS other
- FROM hawq_size_of_table_disk as sotd, pg_class
- WHERE sotd.sotdoid=pg_class.oid ORDER BY relname;
+AS toast, sotdadditionalsize AS other
+FROM hawq_size_of_table_disk as sotd, pg_class
+WHERE sotd.sotdoid=pg_class.oid ORDER BY relname;
```
##### Viewing Disk Space Usage for Indexes <a id="topic19"></a>
The *hawq\_toolkit* administrative schema contains a number of views for checking index sizes. To see the total size of all index\(es\) on a table, use the *hawq\_size\_of\_all\_table\_indexes* view. To see the size of a particular index, use the *hawq\_size\_of\_index* view. The index sizing views list tables and indexes by object ID \(not by name\). To check the size of an index by name, you must look up the relation name \(`relname`\) in the *pg\_class* table. For example:
-```
+```sql
=> SELECT soisize, relname as indexname
- FROM pg_class, hawq_size_of_index
- WHERE pg_class.oid=hawq_size_of_index.soioid
- AND pg_class.relkind='i';
+FROM pg_class, hawq_size_of_index
+WHERE pg_class.oid=hawq_size_of_index.soioid
+AND pg_class.relkind='i';
```
### Viewing Metadata Information about Database Objects <a id="topic24"></a>
@@ -80,12 +80,15 @@
You can use the system views *pg\_stat\_operations* and *pg\_stat\_partition\_operations* to look up actions performed on an object, such as a table. For example, to see the actions performed on a table, such as when it was created and when it was last vacuumed and analyzed:
-```
+```sql
=> SELECT schemaname as schema, objname as table,
- usename as role, actionname as action,
- subtype as type, statime as time
- FROM pg_stat_operations
- WHERE objname='cust';
+usename as role, actionname as action,
+subtype as type, statime as time
+FROM pg_stat_operations
+WHERE objname='cust';
+```
+
+```
schema | table | role | action | type | time
--------+-------+------+---------+-------+--------------------------
sales | cust | main | CREATE | TABLE | 2010-02-09 18:10:07.867977-08
@@ -161,13 +164,13 @@
HAWQ provides a utility called `gplogfilter` can search through a HAWQ log file for entries matching the specified criteria. By default, this utility searches through the HAWQ master log file in the default logging location. For example, to display the entries to the master log file starting after 2 pm on a certain date:
-```
+```shell
$ gplogfilter -b '2016-01-18 14:00'
```
To search through all segment log files simultaneously, run `gplogfilter` through the `hawq ssh` utility. For example, specify the seg\_host\_log\_file that contains hosts to participate in the session, then use `gplogfilter` to display the last three lines of each segment log file:
-```
+```shell
$ hawq ssh -f seg_host_log_file
=> source ~/greenplum_path.sh
=> gplogfilter -n 3 /data/hawq-install-path/segmentdd/pg_log/hawq*.csv
@@ -177,7 +180,7 @@
Use HAWQ's administrative schema *hawq\_toolkit* to query the system catalogs, log files, and operating environment for system status information. The `hawq_toolkit` schema contains several views you can access using SQL commands. The *hawq\_toolkit* schema is accessible to all database users. Some objects require superuser permissions. Use a command similar to the following to add the *hawq\_toolkit* schema to your schema search path:
-```
+```sql
=> ALTER ROLE myrole SET search_path TO myschema,hawq_toolkit;
```
diff --git a/admin/startstop.md b/admin/startstop.md
index 3a24563..fbc86a7 100644
--- a/admin/startstop.md
+++ b/admin/startstop.md
@@ -12,7 +12,7 @@
Do not issue a `KILL` command to end any Postgres process. Instead, use the database command `pg_cancel_backend()`.
-For information about [hawq start](/200/hawq/reference/cli/admin_utilities/hawqstart.html) and [hawq stop](/200/hawq/reference/cli/admin_utilities/hawqstop.html), see the appropriate pages in the HAWQ Management Utility Reference or enter `hawq start -h` or `hawq stop -h` on the command line.
+For information about [hawq start](/20/reference/cli/admin_utilities/hawqstart.html) and [hawq stop](/20/reference/cli/admin_utilities/hawqstop.html), see the appropriate pages in the HAWQ Management Utility Reference or enter `hawq start -h` or `hawq stop -h` on the command line.
## Initialize HAWQ <a id="task_g1y_xtm_s5"></a>
@@ -27,7 +27,7 @@
- Before initializing HAWQ, set the `$GPHOME` environment variable to point to the location of your HAWQ installation on the master host and exchange SSH keys between all host addresses in the array, using `hawq ssh-exkeys`.
- To initialize and start a HAWQ cluster, enter the following command on the master host:
- ```
+ ```shell
$ hawq init cluster
```
@@ -42,7 +42,7 @@
- Run `hawq start cluster` on the master host to start a HAWQ system:
- ```
+ ```shell
$ hawq start cluster
```
@@ -59,7 +59,7 @@
- To restart a HAWQ cluster, enter the following command on the master host:
- ```
+ ```shell
$ hawq restart cluster
```
@@ -68,11 +68,11 @@
Reload changes to the HAWQ configuration files without interrupting the system.
-The `hawq stop` command can reload changes to the pg\_hba.conf configuration file and to *runtime* parameters in the hawq-site.xml file and pg\_hba.conf file without service interruption. Active sessions pick up changes when they reconnect to the database. Many server configuration parameters require a full system restart \(`hawq restart cluster`\) to activate. For information about server configuration parameters, see the [Server Configuration Parameter Reference](/200/hawq/reference/guc/guc_config.html).
+The `hawq stop` command can reload changes to the pg\_hba.conf configuration file and to *runtime* parameters in the hawq-site.xml file and pg\_hba.conf file without service interruption. Active sessions pick up changes when they reconnect to the database. Many server configuration parameters require a full system restart \(`hawq restart cluster`\) to activate. For information about server configuration parameters, see the [Server Configuration Parameter Reference](/20/reference/guc/guc_config.html).
- Reload configuration file changes without shutting down the system using the `hawq stop` command:
- ```
+ ```shell
$ hawq stop --reload
```
@@ -85,18 +85,18 @@
1. Run `hawq start` using the master -m option:
- ```
+ ```shell
$ hawq start master -m
```
2. Connect to the master in maintenance mode to do catalog maintenance. For example:
- ```
+ ```shell
$ PGOPTIONS='-c gp_session_role=utility' psql template1
```
3. After completing your administrative tasks, restart the master in production mode.
- ```
+ ```shell
$ hawq restart master
```
@@ -111,13 +111,13 @@
- To stop HAWQ:
- ```
+ ```shell
$ hawq stop cluster
```
- To stop HAWQ in fast mode:
- ```
+ ```shell
$ hawq stop cluster -M fast
```
@@ -130,26 +130,27 @@
- Stop the entire HAWQ system by stopping the cluster on the master host.
- ```
+ ```shell
$ hawq stop cluster
```
- To stop segments and kill any running queries without causing data loss or inconsistency issues, use `fast` or `immediate` mode on the cluster:
- ```
+ ```shell
$ hawq stop cluster -M fast
$ hawq stop cluster -M immediate
```
- Use `hawq stop master` to stop the master only. If you cannot stop the master due to running transactions, try using fast shutdown. If fast shutdown does not work, use immediate shutdown. Use immediate shutdown with caution, as it will result in a crash-recovery run when the system is restarted.
- ```
+
+ ```shell
$ hawq stop master -M fast
$ hawq stop master -M immediate
```
- When stopping a segment or all segments, you can use the default mode of smart mode. Using fast or immediate mode on segments will have no effect since segments are stateless.
- ```
+ ```shell
$ hawq stop segment
$ hawq stop allsegments
```
diff --git a/bestpractices/HAWQBestPracticesOverview.html.md.erb b/bestpractices/HAWQBestPracticesOverview.html.md.erb
new file mode 100644
index 0000000..d46b87e
--- /dev/null
+++ b/bestpractices/HAWQBestPracticesOverview.html.md.erb
@@ -0,0 +1,31 @@
+---
+title: Best Practices
+---
+
+This chapter provides best practices on using the components and features that are part of a HAWQ system.
+
+- **[HAWQ Best Practices](../bestpractices/general_bestpractices.html)**
+
+ This topic addresses general best practices for users who are new to HAWQ. HAWQ 2.0 consists of substantial changes to architectural components and user interfaces.
+
+- **[Best Practices for Operating HAWQ](../bestpractices/operating_hawq_bestpractices.html)**
+
+ This topic provides best practices for operating HAWQ, including recommendations for stopping, starting and monitoring HAWQ.
+
+- **[Best Practices for Securing HAWQ](../bestpractices/secure_bestpractices.html)**
+
+ To secure your HAWQ deployment, review the recommendations listed in this topic.
+
+- **[Best Practices for Managing Resources](../bestpractices/managing_resources_bestpractices.html)**
+
+ This topic describes best practices for managing resources in HAWQ.
+
+- **[Best Practices for Managing Data](../bestpractices/managing_data_bestpractices.html)**
+
+ This topic describes best practices for creating databases, loading data, partioning data, and recovering data in HAWQ.
+
+- **[Best Practices for Querying Data](../bestpractices/querying_data_bestpractices.html)**
+
+ To obtain the best results when querying data in HAWQ, review the best practices described in this topic.
+
+
diff --git a/bestpractices/general_bestpractices.html.md.erb b/bestpractices/general_bestpractices.html.md.erb
new file mode 100644
index 0000000..c2d575f
--- /dev/null
+++ b/bestpractices/general_bestpractices.html.md.erb
@@ -0,0 +1,23 @@
+---
+title: HAWQ Best Practices
+---
+
+This topic addresses general best practices for users who are new to HAWQ. HAWQ 2.0 consists of substantial changes to architectural components and user interfaces.
+
+When using HAWQ, adhere to the following guidelines for best results:
+
+- **Use a consistent `hawq-site.xml` file to configure your entire cluster**:
+
+ Configuration guc/parameters are located in `$GPHOME/etc/hawq-site.xml`. This configuration file resides on all HAWQ instances and can be modified by using the `hawq config` utility. You can use the same configuration file cluster-wide across both master and segments.
+
+ **Note:** While `postgresql.conf` still exists in HAWQ, any parameters defined in `hawq-site.xml` will overwrite configurations in `postgresql.conf`. For this reason, we recommend that you only use `hawq-site.xml` to configure your HAWQ cluster.
+
+- **Keep in mind the factors that impact the number of virtual segments used for queries. The number of virtual segments used directly impacts the query's performance.** The degree of parallelism achieved by a query is determined by multiple factors, including the following:
+ - **Cost of the query**. Small queries use fewer segments and larger queries use more segments. Note that there are some techniques you can use when defining resource queues to influence the number of virtual segments and general resources that are allocated to queries. See [Best Practices for Using Resource Queues](managing_resources_bestpractices.html#topic_hvd_pls_wv).
+ - **Available resources**. Resources available at query time. If more resources are available in the resource queue, the resources will be used.
+ - **Hash table and bucket number**. If the query involves only hash-distributed tables, and the bucket number (bucketnum) configured for all the hash tables is either the same bucket number for all tables or the table size for random tables is no more than 1.5 times larger than the size of hash tables for the hash tables, then the query's parallelism is fixed (equal to the hash table bucket number). Otherwise, the number of virtual segments depends on the query's cost and hash-distributed table queries will behave like queries on randomly distributed tables.
+ - **Query Type**: For queries with some user-defined functions or for external tables where calculating resource costs is difficult , then the number of virtual segments is controlled by `hawq_rm_nvseg_perquery_limit `and `hawq_rm_nvseg_perquery_perseg_limit` parameters, as well as by the ON clause and the location list of external tables. If the query has a hash result table (e.g. `INSERT into hash_table`) then the number of virtual segment number must be equal to the bucket number of the resulting hash table, If the query is performed in utility mode, such as for `COPY` and `ANALYZE` operations, the virtual segment number is calculated by different policies, which will be explained later in this section.
+
+ See [Query Performance](../query/query-performance.html#topic38) for more details.
+
+
diff --git a/bestpractices/managing_data_bestpractices.html.md.erb b/bestpractices/managing_data_bestpractices.html.md.erb
new file mode 100644
index 0000000..6ac1c4b
--- /dev/null
+++ b/bestpractices/managing_data_bestpractices.html.md.erb
@@ -0,0 +1,47 @@
+---
+title: Best Practices for Managing Data
+---
+
+This topic describes best practices for creating databases, loading data, partioning data, and recovering data in HAWQ.
+
+## Best Practices for Loading Data<a id="topic_xhy_v2j_1v"></a>
+
+Loading data into HDFS is challenging due to the limit on the number of files that can be opened concurrently for write on both NameNodes and DataNodes.
+
+To obtain the best performance during data loading, observe the following best practices:
+
+- Typically the number of concurrent connections to a NameNode should not exceed 50,000, and the number of open files per DataNode should not exceed 10,000. If you exceed these limits, NameNode and DataNode may become overloaded and slow.
+- If the number of partitions in a table is large, the recommended way to load data into the partitioned table is to load the data partition by partition. For example, you can use query such as the following to load data into only one partition:
+
+ ```sql
+ insert into target_partitioned_table_part1 select * from source_table where filter
+ ```
+
+ where *filter* selects only the data in the target partition.
+
+- To alleviate the load on NameNode, you can reduce the number of virtual segment used per node. You can do this on the statement-level or on the resource queue level. See [Configuring the Maximum Number of Virtual Segments](../resourcemgmt/ConfigureResourceManagement.html#topic_tl5_wq1_f5) for more information.
+- Use resource queues to limit load query and read query concurrency.
+
+The best practice for loading data into partitioned tables is to create an intermediate staging table, load it, and then exchange it into your partition design. See [Exchanging a Partition](../ddl/ddl-partition.html#topic83).
+
+## Best Practices for Partitioning Data<a id="topic_s23_52j_1v"></a>
+
+### Deciding on a Table Partitioning Strategy<a id="topic65"></a>
+
+Not all tables are good candidates for partitioning. If the answer is *yes* to all or most of the following questions, table partitioning is a viable database design strategy for improving query performance. If the answer is *no* to most of the following questions, table partitioning is not the right solution for that table. Test your design strategy to ensure that query performance improves as expected.
+
+- **Is the table large enough?** Large fact tables are good candidates for table partitioning. If you have millions or billions of records in a table, you may see performance benefits from logically breaking that data up into smaller chunks. For smaller tables with only a few thousand rows or less, the administrative overhead of maintaining the partitions will outweigh any performance benefits you might see.
+- **Are you experiencing unsatisfactory performance?** As with any performance tuning initiative, a table should be partitioned only if queries against that table are producing slower response times than desired.
+- **Do your query predicates have identifiable access patterns?** Examine the `WHERE` clauses of your query workload and look for table columns that are consistently used to access data. For example, if most of your queries tend to look up records by date, then a monthly or weekly date-partitioning design might be beneficial. Or if you tend to access records by region, consider a list-partitioning design to divide the table by region.
+- **Does your data warehouse maintain a window of historical data?** Another consideration for partition design is your organization's business requirements for maintaining historical data. For example, your data warehouse may require that you keep data for the past twelve months. If the data is partitioned by month, you can easily drop the oldest monthly partition from the warehouse and load current data into the most recent monthly partition.
+- **Can the data be divided into somewhat equal parts based on some defining criteria?** Choose partitioning criteria that will divide your data as evenly as possible. If the partitions contain a relatively equal number of records, query performance improves based on the number of partitions created. For example, by dividing a large table into 10 partitions, a query will execute 10 times faster than it would against the unpartitioned table, provided that the partitions are designed to support the query's criteria.
+
+Do not create more partitions than are needed. Creating too many partitions can slow down management and maintenance jobs, such as vacuuming, recovering segments, expanding the cluster, checking disk usage, and others.
+
+Partitioning does not improve query performance unless the query optimizer can eliminate partitions based on the query predicates. Queries that scan every partition run slower than if the table were not partitioned, so avoid partitioning if few of your queries achieve partition elimination. Check the explain plan for queries to make sure that partitions are eliminated. See [Query Profiling](../query/query-profiling.html#topic39) for more about partition elimination.
+
+Be very careful with multi-level partitioning because the number of partition files can grow very quickly. For example, if a table is partitioned by both day and city, and there are 1,000 days of data and 1,000 cities, the total number of partitions is one million. Column-oriented tables store each column in a physical table, so if this table has 100 columns, the system would be required to manage 100 million files for the table.
+
+Before settling on a multi-level partitioning strategy, consider a single level partition with bitmap indexes. Indexes slow down data loads, so consider performance testing with your data and schema to decide on the best strategy.
+
+
diff --git a/bestpractices/managing_resources_bestpractices.html.md.erb b/bestpractices/managing_resources_bestpractices.html.md.erb
new file mode 100644
index 0000000..7cb6e9d
--- /dev/null
+++ b/bestpractices/managing_resources_bestpractices.html.md.erb
@@ -0,0 +1,144 @@
+---
+title: Best Practices for Managing Resources
+---
+
+This topic describes best practices for managing resources in HAWQ.
+
+## Best Practices for Configuring Resource Management<a id="topic_ikz_ndx_15"></a>
+
+When configuring resource management, you can apply certain best practices to ensure that resources are managed both efficiently and for best system performance.
+
+The following is a list of high-level best practices for optimal resource management:
+
+- Make sure segments do not have identical IP addresses. See [Segments Do Not Appear in gp\_segment\_configuration](../troubleshooting/Troubleshooting.html#topic_hlj_zxx_15) for an explanation of this problem.
+- Configure all segments to have the same resource capacity. See [Configuring Segment Resource Capacity](../resourcemgmt/ConfigureResourceManagement.html#topic_htk_fxh_15).
+- To prevent resource fragmentation, ensure that your deployment's segment resource capacity (standalone mode) or YARN node resource capacity (YARN mode) is a multiple of all virtual segment resource quotas. See [Configuring Segment Resource Capacity](../resourcemgmt/ConfigureResourceManagement.html#topic_htk_fxh_15) (HAWQ standalone mode) and [Setting HAWQ Segment Resource Capacity in YARN](../resourcemgmt/YARNIntegration.html#topic_pzf_kqn_c5).
+- Ensure that enough registered segments are available and usable for query resource requests. If the number of unavailable or unregistered segments is higher than a set limit, then query resource requests are rejected. Also ensure that the variance of dispatched virtual segments across physical segments is not greater than the configured limit. See [Rejection of Query Resource Requests](../troubleshooting/Troubleshooting.html#topic_vm5_znx_15).
+- Use multiple master and segment temporary directories on separate, large disks (2TB or greater) to load balance writes to temporary files (for example, `/disk1/tmp /disk2/tmp`). For a given query, HAWQ will use a separate temp directory (if available) for each virtual segment to store spill files. Multiple HAWQ sessions will also use separate temp directories where available to avoid disk contention. If you configure too few temp directories, or you place multiple temp directories on the same disk, you increase the risk of disk contention or running out of disk space when multiple virtual segments target the same disk.
+- Configure minimum resource levels in YARN, and tune the timeout of when idle resources are returned to YARN. See [Tune HAWQ Resource Negotiations with YARN](../resourcemgmt/YARNIntegration.html#topic_wp3_4bx_15).
+- Make sure that the property `yarn.scheduler.minimum-allocation-mb` in `yarn-site.xml` is an equal subdivision of 1GB. For example, 1024, 512.
+
+## Best Practices for Using Resource Queues<a id="topic_hvd_pls_wv"></a>
+
+Design and configure your resource queues depending on the operational needs of your deployment. This topic describes the best practices for creating and modifying resource queues within the context of different operational scenarios.
+
+### Modifying Resource Queues for Overloaded HDFS
+
+A high number of concurrent HAWQ queries can cause HDFS to overload, especially when querying partitioned tables. Use the `ACTIVE_STATEMENTS` attribute to restrict statement concurrency in a resource queue. For example, if an external application is executing more than 100 concurrent queries, then limiting the number of active statements in your resource queues will instruct the HAWQ resource manager to restrict actual statement concurrency within HAWQ. You might want to modify an existing resource queue as follows:
+
+```sql
+ALTER RESOURCE QUEUE sampleque1 WITH (ACTIVE_STATEMENTS=20);
+```
+
+In this case, when this DDL is applied to queue `sampleque1`, the roles using this queue will have to wait until no more than 20 statements are running to execute their queries. Therefore, 80 queries will be waiting in the queue for later execution. Restricting the number of active query statements helps limit the usage of HDFS resources and protects HDFS. You can alter concurrency even when the resource queue is busy. For example, if a queue already has 40 concurrent statements running, and you apply a DDL statement that specifies `ACTIVE_STATEMENTS=20`, then the resource queue pauses the allocation of resources to queries until more than 20 statements have returned their resources.
+
+### Isolating and Protecting Production Workloads
+
+Another best practice is using resource queues to isolate your workloads. Workload isolation prevents your production workload from being starved of resources. To create this isolation, divide your workload by creating roles for specific purposes. For example, you could create one role for production online verification and another role for the regular running of production processes.
+
+In this scenario, let us assign `role1` for the production workload and `role2` for production software verification. We can define the following resource queues under the same parent queue `dept1que`, which is the resource queue defined for the entire department.
+
+```sql
+CREATE RESOURCE QUEUE dept1product
+ WITH (PARENT='dept1que', MEMORY_LIMIT_CLUSTER=90%, CORE_LIMIT_CLUSTER=90%, RESOURCE_OVERCOMMIT_FACTOR=2);
+
+CREATE RESOURCE QUEUE dept1verification
+ WITH (PARENT='dept1que', MEMORY_LIMIT_CLUSTER=10%, CORE_LIMIT_CLUSTER=10%, RESOURCE_OVERCOMMIT_FACTOR=10);
+
+ALTER ROLE role1 RESOURCE QUEUE dept1product;
+
+ALTER ROLE role2 RESOURCE QUEUE dept1verification;
+```
+
+With these resource queues defined, workload is spread across the resource queues as follows:
+
+- When both `role1` and `role2` have workloads, the test verification workload gets only 10% of the total available `dept1que` resources, leaving 90% of the `dept1que` resources available for running the production workload.
+- When `role1` has a workload but `role2` is idle, then 100% of all `dept1que` resources can be consumed by the production workload.
+- When only `role2` has a workload (for example, during a scheduled testing window), then 100% of all `dept1que` resources can also be utilized for testing.
+
+Even when the resource queues are busy, you can alter the resource queue's memory and core limits to change resource allocation policies before switching workloads.
+
+In addition, you can use resource queues to isolate workloads for different departments or different applications. For example, we can use the following DDL statements to define 3 departments, and an administrator can arbitrarily redistribute resource allocations among the departments according to usage requirements.
+
+```sql
+ALTER RESOURCE QUEUE pg_default
+ WITH (MEMORY_LIMIT_CLUSTER=10%, CORE_LIMIT_CLUSTER=10%);
+
+CREATE RESOURCE QUEUE dept1
+ WITH (PARENT='pg_root', MEMORY_LIMIT_CLUSTER=30%, CORE_LIMIT_CLUSTER=30%);
+
+CREATE RESOURCE QUEUE dept2
+ WITH (PARENT='pg_root', MEMORY_LIMIT_CLUSTER=30%, CORE_LIMIT_CLUSTER=30%);
+
+CREATE RESOURCE QUEUE dept3
+ WITH (PARENT='pg_root', MEMORY_LIMIT_CLUSTER=30%, CORE_LIMIT_CLUSTER=30%);
+
+CREATE RESOURCE QUEUE dept11
+ WITH (PARENT='dept1', MEMORY_LIMIT_CLUSTER=50%,CORE_LIMIT_CLUSTER=50%);
+
+CREATE RESOURCE QUEUE dept12
+ WITH (PARENT='dept1', MEMORY_LIMIT_CLUSTER=50%, CORE_LIMIT_CLUSTER=50%);
+```
+
+### Querying Parquet Tables with Large Table Size
+
+You can use resource queues to improve query performance on Parquet tables with a large page size. This type of query requires a large memory quota for virtual segments. Therefore, if one role mostly queries Parquet tables with a large page size, alter the resource queue associated with the role to increase its virtual segment resource quota. For example:
+
+```sql
+ALTER RESOURCE queue1 WITH (VSEG_RESOURCE_QUOTA='mem:2gb');
+```
+
+If there are only occasional queries on Parquet tables with a large page size, use a statement level specification instead of altering the resource queue. For example:
+
+```sql
+SET HAWQ_RM_STMT_NVSEG=10;
+SET HAWQ_RM_STMT_VSEG_MEMORY='2gb';
+query1;
+SET HAWQ_RM_STMT_NVSEG=0;
+```
+
+### Restricting Resource Consumption for Specific Queries
+
+In general, the HAWQ resource manager attempts to provide as much resources as possible to the current query to achieve high query performance. When a query is complex and large, however, the associated resource queue can use up many virtual segments causing other resource queues (and queries) to starve. Under these circumstances,you should enable nvseg limits on the resource queue associated with the large query. For example, you can specify that all queries can use no more than 200 virtual segments. To achieve this limit, alter the resource queue as follows
+
+``` sql
+ALTER RESOURCE QUEUE queue1 WITH (NVSEG_UPPER_LIMIT=200);
+```
+
+If we hope to make this limit vary according to the dynamic cluster size, we can use the following statement.
+
+```sql
+ALTER RESOURCE QUEUE queue1 WITH (NVSEG_UPPER_LIMIT_PERSEG=10);
+```
+
+After setting the limit in the above example, the actual limit will be 100 if you have a 10-node cluster. If the cluster is expanded to 20 nodes, then the limit increases automatically to 200.
+
+### Guaranteeing Resource Allocations for Individual Statements
+
+In general, the minimum number of virtual segments allocated to a statement is decided by the resource queue's actual capacity and its concurrency setting. For example, if there are 10 nodes in a cluster and the total resource capacity of the cluster is 640GB and 160 cores, then a resource queue having 20% capacity has a capacity of 128GB (640GB \* .20) and 32 cores (160 \*.20). If the virtual segment quota is set to 256MB, then this queue has 512 virtual segments allocated (128GB/256MB=512). If the `ACTIVE_STATEMENTS` concurrency setting for the resource queue is 20, then the minimum number of allocated virtual segments for each query is **25** (*trunc*(512/20)=25). However, this minimum number of virtual segments is a soft restriction. If a query statement requires only 5 virtual segments, then this minimum number of 25 is ignored since it is not necessary to allocate 25 for this statement.
+
+In order to raise the minimum number of virtual segments available for a query statement, there are two options.
+
+- *Option 1*: Alter the resource queue to reduce concurrency. This is the recommended way to achieve the goal. For example:
+
+ ```sql
+ ALTER RESOURCE QUEUE queue1 WITH (ACTIVE_STATEMENTS=10);
+ ```
+
+ If the original concurrency setting is 20, then the minimum number of virtual segments is doubled.
+
+- *Option 2*: Alter the nvseg limits of the resource queue. For example:
+
+ ```sql
+ ALTER RESOURCE QUEUE queue1 WITH (NVSEG_LOWER_LIMIT=50);
+ ```
+
+ or, alternately:
+
+ ```sql
+ ALTER RESOURCE QUEUE queue1 WITH (NVSEG_LOWER_LIMIT_PERSEG=5);
+ ```
+
+ In the second DDL, if there are 10 nodes in the cluster, the actual minimum number of virtual segments is 50 (5 \* 10 = 50).
+
+
diff --git a/bestpractices/operating_hawq_bestpractices.html.md.erb b/bestpractices/operating_hawq_bestpractices.html.md.erb
new file mode 100644
index 0000000..f95d895
--- /dev/null
+++ b/bestpractices/operating_hawq_bestpractices.html.md.erb
@@ -0,0 +1,289 @@
+---
+title: Best Practices for Operating HAWQ
+---
+
+This topic provides best practices for operating HAWQ, including recommendations for stopping, starting and monitoring HAWQ.
+
+## Best Practices to Start/Stop HAWQ Cluster Members<a id="task_qgk_bz3_1v"></a>
+
+For best results in using `hawq start` and `hawq stop` to manage your HAWQ system, the following best practices are recommended.
+
+- Issue the `CHECKPOINT` command to update and flush all data files to disk and update the log file before stopping the cluster. A checkpoint ensures that, in the event of a crash, files can be restored from the checkpoint snapshot.
+- Stop the entire HAWQ system by stopping the cluster on the master host:
+ ```shell
+ $ hawq stop cluster
+ ```
+
+- To stop segments and kill any running queries without causing data loss or inconsistency issues, use `fast` or `immediate` mode on the cluster:
+
+ ```shell
+ $ hawq stop cluster -M fast
+ ```
+ ```shell
+ $ hawq stop cluster -M immediate
+ ```
+
+- Use `hawq stop master` to stop the master only. If you cannot stop the master due to running transactions, try using fast shutdown. If fast shutdown does not work, use immediate shutdown. Use immediate shutdown with caution, as it will result in a crash-recovery run when the system is restarted.
+
+ ```shell
+ $ hawq stop master -M fast
+ ```
+ ```shell
+ $ hawq stop master -M immediate
+ ```
+
+- When stopping a segment or all segments, you can use the default mode of smart mode. Using fast or immediate mode on segments will have no effect since segments are stateless.
+
+ ```shell
+ $ hawq stop segment
+ ```
+ ```shell
+ $ hawq stop allsegments
+ ```
+
+- Typically you should always use `hawq start cluster` or `hawq restart cluster` to start the cluster. If you do end up using `hawq start standby|master|segment` to start nodes individually, make sure you always start the standby before the active master. Otherwise, the standby can become unsynchronized with the active master.
+
+## Guidelines for Cluster Expansion<a id="id_trr_m1j_1v"></a>
+
+This topic provides some guidelines around expanding your HAWQ cluster.
+
+There are several recommendations to keep in mind when modifying the size of your running HAWQ cluster:
+
+- When you add a new node, install both a DataNode and a physical segment on the new node.
+- After adding a new node, you should always rebalance HDFS data to maintain cluster performance.
+- Adding or removing a node also necessitates an update to the HDFS metadata cache. This update will happen eventually, but can take some time. To speed the update of the metadata cache, execute **`select gp_metadata_cache_clear();`**.
+- Note that for hash distributed tables, expanding the cluster will not immediately improve performance since hash distributed tables use a fixed number of virtual segments. In order to obtain better performance with hash distributed tables, you must redistribute the table to the updated cluster by either the [ALTER TABLE](../reference/sql/ALTER-TABLE.html) or [CREATE TABLE AS](../reference/sql/CREATE-TABLE-AS.html#topic1) command.
+- If you are using hash tables, consider updating the `default_hash_table_bucket_number` server configuration parameter to a larger value after expanding the cluster but before redistributing the hash tables.
+
+## Database State Monitoring Activities<a id="id_o5n_p1j_1v"></a>
+
+<a id="id_o5n_p1j_1v__d112e31"></a>
+
+<table>
+<caption><span class="tablecap">Table 1. Database State Monitoring Activities</span></caption>
+<colgroup>
+<col width="33%" />
+<col width="33%" />
+<col width="33%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Activity</th>
+<th>Procedure</th>
+<th>Corrective Actions</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>List segments that are currently down. If any rows are returned, this should generate a warning or alert.
+<p>Recommended frequency: run every 5 to 10 minutes</p>
+<p>Severity: IMPORTANT</p></td>
+<td>Run the following query in the <code class="ph codeph">postgres</code> database:
+<pre class="pre codeblock"><code>SELECT * FROM gp_segment_configuration
+WHERE status <> 'u';</code></pre></td>
+<td>If the query returns any rows, follow these steps to correct the problem:
+<ol>
+<li>Verify that the hosts with down segments are responsive.</li>
+<li>If hosts are OK, check the <span class="ph filepath">pg_log</span> files for the down segments to discover the root cause of the segments going down.</li>
+<li>If no unexpected errors are found, run the <code class="ph codeph">gprecoverseg</code> utility to bring the segments back online.</li>
+</ol></td>
+</tr>
+</tbody>
+</table>
+
+
+## Hardware and Operating System Monitoring<a id="id_d3w_p1j_1v"></a>
+
+<a id="id_d3w_p1j_1v__d112e111"></a>
+
+<table>
+<caption><span class="tablecap">Table 2. Hardware and Operating System Monitoring Activities</span></caption>
+<colgroup>
+<col width="33%" />
+<col width="33%" />
+<col width="33%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Activity</th>
+<th>Procedure</th>
+<th>Corrective Actions</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>Underlying platform check for maintenance required or system down of the hardware.
+<p>Recommended frequency: real-time, if possible, or every 15 minutes</p>
+<p>Severity: CRITICAL</p></td>
+<td>Set up system check for hardware and OS errors.</td>
+<td>If required, remove a machine from the HAWQ cluster to resolve hardware and OS issues, then, after add it back to the cluster and run <code class="ph codeph">gprecoverseg</code>.</td>
+</tr>
+<tr class="even">
+<td>Check disk space usage on volumes used for HAWQ data storage and the OS.
+<p>Recommended frequency: every 5 to 30 minutes</p>
+<p>Severity: CRITICAL</p></td>
+<td><div class="p">
+Set up a disk space check.
+<ul>
+<li>Set a threshold to raise an alert when a disk reaches a percentage of capacity. The recommended threshold is 75% full.</li>
+<li>It is not recommended to run the system with capacities approaching 100%.</li>
+</ul>
+</div></td>
+<td>Free space on the system by removing some data or files.</td>
+</tr>
+<tr class="odd">
+<td>Check for errors or dropped packets on the network interfaces.
+<p>Recommended frequency: hourly</p>
+<p>Severity: IMPORTANT</p></td>
+<td>Set up a network interface checks.</td>
+<td><p>Work with network and OS teams to resolve errors.</p></td>
+</tr>
+<tr class="even">
+<td>Check for RAID errors or degraded RAID performance.
+<p>Recommended frequency: every 5 minutes</p>
+<p>Severity: CRITICAL</p></td>
+<td>Set up a RAID check.</td>
+<td><ul>
+<li>Replace failed disks as soon as possible.</li>
+<li>Work with system administration team to resolve other RAID or controller errors as soon as possible.</li>
+</ul></td>
+</tr>
+<tr class="odd">
+<td>Check for adequate I/O bandwidth and I/O skew.
+<p>Recommended frequency: when create a cluster or when hardware issues are suspected.</p></td>
+<td>Run the HAWQ <code class="ph codeph">hawq checkperf</code> utility.</td>
+<td><div class="p">
+The cluster may be under-specified if data transfer rates are not similar to the following:
+<ul>
+<li>2GB per second disk read</li>
+<li>1 GB per second disk write</li>
+<li>10 Gigabit per second network read and write</li>
+</ul>
+If transfer rates are lower than expected, consult with your data architect regarding performance expectations.
+</div>
+<p>If the machines on the cluster display an uneven performance profile, work with the system administration team to fix faulty machines.</p></td>
+</tr>
+</tbody>
+</table>
+
+
+## Data Maintenance<a id="id_khd_q1j_1v"></a>
+
+<a id="id_khd_q1j_1v__d112e279"></a>
+
+<table>
+<caption><span class="tablecap">Table 3. Data Maintenance Activities</span></caption>
+<colgroup>
+<col width="33%" />
+<col width="33%" />
+<col width="33%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Activity</th>
+<th>Procedure</th>
+<th>Corrective Actions</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>Check for missing statistics on tables.</td>
+<td>Check the <code class="ph codeph">hawq_stats_missing</code> view in each database:
+<pre class="pre codeblock"><code>SELECT * FROM hawq_toolkit.hawq_stats_missing;</code></pre></td>
+<td>Run <code class="ph codeph">ANALYZE</code> on tables that are missing statistics.</td>
+</tr>
+</tbody>
+</table>
+
+
+## Database Maintenance<a id="id_lx4_q1j_1v"></a>
+
+<a id="id_lx4_q1j_1v__d112e343"></a>
+
+<table>
+<caption><span class="tablecap">Table 4. Database Maintenance Activities</span></caption>
+<colgroup>
+<col width="33%" />
+<col width="33%" />
+<col width="33%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Activity</th>
+<th>Procedure</th>
+<th>Corrective Actions</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>Mark deleted rows in HAWQ system catalogs (tables in the <code class="ph codeph">pg_catalog</code> schema) so that the space they occupy can be reused.
+<p>Recommended frequency: daily</p>
+<p>Severity: CRITICAL</p></td>
+<td>Vacuum each system catalog:
+<pre class="pre codeblock"><code>VACUUM <table>;</code></pre></td>
+<td>Vacuum system catalogs regularly to prevent bloating.</td>
+</tr>
+<tr class="even">
+<td>Update table statistics.
+<p>Recommended frequency: after loading data and before executing queries</p>
+<p>Severity: CRITICAL</p></td>
+<td>Analyze user tables:
+<pre class="pre codeblock"><code>ANALYZEDB -d <database> -a</code></pre></td>
+<td>Analyze updated tables regularly so that the optimizer can produce efficient query execution plans.</td>
+</tr>
+<tr class="odd">
+<td>Backup the database data.
+<p>Recommended frequency: daily, or as required by your backup plan</p>
+<p>Severity: CRITICAL</p></td>
+<td>See <a href="../admin/BackingUpandRestoringHAWQDatabases.html">Backing up and Restoring HAWQ Databases</a> for a discussion of backup procedures</td>
+<td>Best practice is to have a current backup ready in case the database must be restored.</td>
+</tr>
+<tr class="even">
+<td>Reindex system catalogs (tables in the <code class="ph codeph">pg_catalog</code> schema) to maintain an efficient catalog.
+<p>Recommended frequency: weekly, or more often if database objects are created and dropped frequently</p></td>
+<td>Run <code class="ph codeph">REINDEX SYSTEM</code> in each database.
+<pre class="pre codeblock"><code>REINDEXDB -s</code></pre></td>
+<td>The optimizer retrieves information from the system tables to create query plans. If system tables and indexes are allowed to become bloated over time, scanning the system tables increases query execution time.</td>
+</tr>
+</tbody>
+</table>
+
+
+## Patching and Upgrading<a id="id_blv_q1j_1v"></a>
+
+<a id="id_blv_q1j_1v__d112e472"></a>
+
+<table>
+<caption><span class="tablecap">Table 5. Patch and Upgrade Activities</span></caption>
+<colgroup>
+<col width="33%" />
+<col width="33%" />
+<col width="33%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Activity</th>
+<th>Procedure</th>
+<th>Corrective Actions</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>Ensure any bug fixes or enhancements are applied to the kernel.
+<p>Recommended frequency: at least every 6 months</p>
+<p>Severity: IMPORTANT</p></td>
+<td>Follow the vendor's instructions to update the Linux kernel.</td>
+<td>Keep the kernel current to include bug fixes and security fixes, and to avoid difficult future upgrades.</td>
+</tr>
+<tr class="even">
+<td>Install HAWQ minor releases.
+<p>Recommended frequency: quarterly</p>
+<p>Severity: IMPORTANT</p></td>
+<td>Always upgrade to the latest in the series.</td>
+<td>Keep the HAWQ software current to incorporate bug fixes, performance enhancements, and feature enhancements into your HAWQ cluster.</td>
+</tr>
+</tbody>
+</table>
+
+
+
diff --git a/bestpractices/querying_data_bestpractices.html.md.erb b/bestpractices/querying_data_bestpractices.html.md.erb
new file mode 100644
index 0000000..75d7100
--- /dev/null
+++ b/bestpractices/querying_data_bestpractices.html.md.erb
@@ -0,0 +1,25 @@
+---
+title: Best Practices for Querying Data
+---
+
+To obtain the best results when querying data in HAWQ, review the best practices described in this topic.
+
+## Examining Query Plans to Solve Problems<a id="id_xtk_jmq_1v"></a>
+
+If a query performs poorly, examine its query plan and ask the following questions:
+
+- **Do operations in the plan take an exceptionally long time?** Look for an operation that consumes the majority of query processing time. For example, if a scan on a hash table takes longer than expected, the data locality may be low; reloading the data can increase the data locality and speed up the query. Or, adjust `enable_<operator>` parameters to see if you can force the legacy query optimizer (planner) to choose a different plan by disabling a particular query plan operator for that query.
+- **Are the optimizer's estimates close to reality?** Run `EXPLAIN ANALYZE` and see if the number of rows the optimizer estimates is close to the number of rows the query operation actually returns. If there is a large discrepancy, collect more statistics on the relevant columns.
+- **Are selective predicates applied early in the plan?** Apply the most selective filters early in the plan so fewer rows move up the plan tree. If the query plan does not correctly estimate query predicate selectivity, collect more statistics on the relevant columns. You can also try reordering the `WHERE` clause of your SQL statement.
+- **Does the optimizer choose the best join order?** When you have a query that joins multiple tables, make sure that the optimizer chooses the most selective join order. Joins that eliminate the largest number of rows should be done earlier in the plan so fewer rows move up the plan tree.
+
+ If the plan is not choosing the optimal join order, set `join_collapse_limit=1` and use explicit `JOIN` syntax in your SQL statement to force the legacy query optimizer (planner) to the specified join order. You can also collect more statistics on the relevant join columns.
+
+- **Does the optimizer selectively scan partitioned tables?** If you use table partitioning, is the optimizer selectively scanning only the child tables required to satisfy the query predicates? Scans of the parent tables should return 0 rows since the parent tables do not contain any data. See [Verifying Your Partition Strategy](../ddl/ddl-partition.html#topic74) for an example of a query plan that shows a selective partition scan.
+- **Does the optimizer choose hash aggregate and hash join operations where applicable?** Hash operations are typically much faster than other types of joins or aggregations. Row comparison and sorting is done in memory rather than reading/writing from disk. To enable the query optimizer to choose hash operations, there must be sufficient memory available to hold the estimated number of rows. Try increasing work memory to improve performance for a query. If possible, run an `EXPLAIN ANALYZE` for the query to show which plan operations spilled to disk, how much work memory they used, and how much memory was required to avoid spilling to disk. For example:
+
+ `Work_mem used: 23430K bytes avg, 23430K bytes max (seg0). Work_mem wanted: 33649K bytes avg, 33649K bytes max (seg0) to lessen workfile I/O affecting 2 workers.`
+
+ The "bytes wanted" message from `EXPLAIN ANALYZE` is based on the amount of data written to work files and is not exact. The minimum `work_mem` needed can differ from the suggested value.
+
+
diff --git a/bestpractices/secure_bestpractices.html.md.erb b/bestpractices/secure_bestpractices.html.md.erb
new file mode 100644
index 0000000..04c5343
--- /dev/null
+++ b/bestpractices/secure_bestpractices.html.md.erb
@@ -0,0 +1,11 @@
+---
+title: Best Practices for Securing HAWQ
+---
+
+To secure your HAWQ deployment, review the recommendations listed in this topic.
+
+- Set up SSL to encrypt your client server communication channel. See [Encrypting Client/Server Connections](../clientaccess/client_auth.html#topic5).
+- Configure `pg_hba.conf` only on HAWQ master. Do not configure it on segments.
+ **Note:** For a more secure system, consider removing all connections that use trust authentication from your master `pg_hba.conf`. Trust authentication means the role is granted access without any authentication, therefore bypassing all security. Replace trust entries with ident authentication if your system has an ident service available.
+
+
diff --git a/clientaccess/kerberos.md b/clientaccess/kerberos.md
index 1f2b5ed..afc4aaa 100644
--- a/clientaccess/kerberos.md
+++ b/clientaccess/kerberos.md
@@ -229,7 +229,7 @@
3. Modify the HAWQ file `pg_hba.conf` to enable Kerberos support. Then restart HAWQ \(`hawq restart -a`\). For example, adding the following line to `pg_hba.conf` adds GSSAPI and Kerberos support. The value for `krb_realm` is the Kerberos realm that is used for authentication to HAWQ.
```
- host all all 0.0.0.0/0 gss include_realm=0 krb_realm=KRB.PIVOTAL.COM
+ host all all 0.0.0.0/0 gss include_realm=0 krb_realm=KRB.EXAMPLE.COM
```
For information about the `pg_hba.conf` file, see [The pg\_hba.conf file](http://www.postgresql.org/docs/8.4/static/auth-pg-hba-conf.html) in the Postgres documentation.
diff --git a/clientaccess/ldap.md b/clientaccess/ldap.md
index d45f5e2..27b204f 100644
--- a/clientaccess/ldap.md
+++ b/clientaccess/ldap.md
@@ -18,29 +18,29 @@
To enable STARTTLS with the TLS protocol, specify the `ldaptls` parameter with the value 1. The default port is 389. In this example, the authentication method parameters include the `ldaptls` parameter.
```
-ldap ldapserver=myldap.com ldaptls=1 ldapprefix="uid=" ldapsuffix=",ou=People,dc=pivotal,dc=com"
+ldap ldapserver=ldap.example.com ldaptls=1 ldapprefix="uid=" ldapsuffix=",ou=People,dc=example,dc=com"
```
Specify a non-default port, with the `ldapport` parameter. In this example, the authentication method includes the `ldaptls` parameter and the `ldapport` parameter to specify the port 550.
```
-ldap ldapserver=myldap.com ldaptls=1 ldapport=500 ldapprefix="uid=" ldapsuffix=",ou=People,dc=pivotal,dc=com"
+ldap ldapserver=ldap.example.com ldaptls=1 ldapport=500 ldapprefix="uid=" ldapsuffix=",ou=People,dc=example,dc=com"
```
## Enabling LDAP Authentication with a Secure Connection and TLS/SSL
To enable a secure connection with TLS/SSL, add `ldaps://` as the prefix to the LDAP server name specified in the `ldapserver` parameter. The default port is 636.
-This example `ldapserver` parameter specifies a secure connection and the TLS/SSL protocol for the LDAP server `myldap.com`.
+This example `ldapserver` parameter specifies a secure connection and the TLS/SSL protocol for the LDAP server `ldap.example.com`.
```
-ldapserver=ldaps://myldap.com
+ldapserver=ldaps://ldap.example.com
```
To specify a non-default port, add a colon \(:\) and the port number after the LDAP server name. This example `ldapserver` parameter includes the `ldaps://` prefix and the non-default port 550.
```
-ldapserver=ldaps://myldap.com:550
+ldapserver=ldaps://ldap.example.com:550
```
### Notes
@@ -56,7 +56,7 @@
If you have a system-wide OpenLDAP system and logins are configured to use LDAP with TLS or SSL in the pg_hba.conf file, logins may fail with the following message:
-```
+```shell
could not start LDAP TLS session: error code '-11'
```
@@ -79,8 +79,8 @@
```
SASL_NOCANON on
- URI ldaps://ldapA.pivotal.priv ldaps://ldapB.pivotal.priv ldaps://ldapC.pivotal.priv
- BASE dc=pivotal,dc=priv
+ URI ldaps://ldapA.example.priv ldaps://ldapB.example.priv ldaps://ldapC.example.priv
+ BASE dc=example,dc=priv
TLS_CACERTDIR /etc/pki/tls/certs
TLS_CACERT /etc/pki/tls/certs/<ca-certificate-file>
```
@@ -89,7 +89,7 @@
4. As the gpadmin user, edit `/usr/local/hawq/greenplum_path.sh` and add the following line.
- ```
+ ```bash
export LDAPCONF=/etc/openldap/ldap.conf
```
@@ -100,17 +100,17 @@
This example specifies LDAP authentication with no encryption between HAWQ and the LDAP server.
```
-host all plainuser 0.0.0.0/0 ldap ldapserver=myldap.com ldapprefix="uid=" ldapsuffix=",ou=People,dc=pivotal,dc=com"
+host all plainuser 0.0.0.0/0 ldap ldapserver=ldap.example.com ldapprefix="uid=" ldapsuffix=",ou=People,dc=example,dc=com"
```
This example specifies LDAP authentication with the STARTTLS and TLS protocol between HAWQ and the LDAP server.
```
-host all tlsuser 0.0.0.0/0 ldap ldapserver=myldap.com ldaptls=1 ldapprefix="uid=" ldapsuffix=",ou=People,dc=pivotal,dc=com"
+host all tlsuser 0.0.0.0/0 ldap ldapserver=ldap.example.com ldaptls=1 ldapprefix="uid=" ldapsuffix=",ou=People,dc=example,dc=com"
```
This example specifies LDAP authentication with a secure connection and TLS/SSL protocol between HAWQ and the LDAP server.
```
-host all ldapsuser 0.0.0.0/0 ldap ldapserver=ldaps://myldap.com ldapprefix="uid=" ldapsuffix=",ou=People,dc=pivotal,dc=com"
+host all ldapsuser 0.0.0.0/0 ldap ldapserver=ldaps://ldap.example.com ldapprefix="uid=" ldapsuffix=",ou=People,dc=example,dc=com"
```
diff --git a/clientaccess/roles_privs.md b/clientaccess/roles_privs.md
index 4638951..f5ac559 100644
--- a/clientaccess/roles_privs.md
+++ b/clientaccess/roles_privs.md
@@ -141,7 +141,7 @@
## Encrypting Data <a id="topic8"></a>
-PostgreSQL provides an optional package of encryption/decryption functions called `pgcrypto`, which can also be installed and used in HAWQ. The `pgcrypto` package is not installed by default with HAWQ, however you can download a `pgcrypto` package from [Pivotal Network](https://network.gopivotal.com/products), then use the Package Manager \(`gppkg`\) to install `pgcrypto` across your entire cluster .
+PostgreSQL provides an optional package of encryption/decryption functions called `pgcrypto`, which can also be installed and used in HAWQ. The `pgcrypto` package is not installed by default with HAWQ. However, you can download a `pgcrypto` package from [Pivotal Network](https://network.pivotal.io) and use the Package Manager \(`gppkg`\) to install `pgcrypto` across your entire cluster. If you are building HAWQ from open source files, then you can enable `pgcrypto` support at build time.
The `pgcrypto` functions allow database administrators to store certain columns of data in encrypted form. This adds an extra layer of protection for sensitive data, as data stored in HAWQ in encrypted form cannot be read by users who do not have the encryption key, nor be read directly from the disks.
diff --git a/datamgmt/BasicDataOperations.html.md.erb b/datamgmt/BasicDataOperations.html.md.erb
new file mode 100644
index 0000000..688e238
--- /dev/null
+++ b/datamgmt/BasicDataOperations.html.md.erb
@@ -0,0 +1,62 @@
+---
+title: Basic Data Operations
+---
+
+This topic describes basic data operations that you perform in HAWQ.
+
+## Inserting Rows<a id="topic3"></a>
+
+Use the `INSERT` command to create rows in a table. This command requires the table name and a value for each column in the table; you may optionally specify the column names in any order. If you do not specify column names, list the data values in the order of the columns in the table, separated by commas.
+
+For example, to specify the column names and the values to insert:
+
+``` sql
+INSERT INTO products (name, price, product_no) VALUES ('Cheese', 9.99, 1);
+```
+
+To specify only the values to insert:
+
+``` sql
+INSERT INTO products VALUES (1, 'Cheese', 9.99);
+```
+
+Usually, the data values are literals (constants), but you can also use scalar expressions. For example:
+
+``` sql
+INSERT INTO films SELECT * FROM tmp_films WHERE date_prod <
+'2004-05-07';
+```
+
+You can insert multiple rows in a single command. For example:
+
+``` sql
+INSERT INTO products (product_no, name, price) VALUES
+ (1, 'Cheese', 9.99),
+ (2, 'Bread', 1.99),
+ (3, 'Milk', 2.99);
+```
+
+To insert data into a partitioned table, you specify the root partitioned table, the table created with the `CREATE TABLE` command. You also can specify a leaf child table of the partitioned table in an `INSERT` command. An error is returned if the data is not valid for the specified leaf child table. Specifying a child table that is not a leaf child table in the `INSERT` command is not supported.
+
+To insert large amounts of data, use external tables or the `COPY` command. These load mechanisms are more efficient than `INSERT` for inserting large quantities of rows. See [Loading and Unloading Data](load/g-loading-and-unloading-data.html#topic1) for more information about bulk data loading.
+
+## Vacuuming the Database<a id="topic9"></a>
+
+Deleted or updated data rows occupy physical space on disk even though new transactions cannot see them. Periodically running the `VACUUM` command removes these expired rows. For example:
+
+``` sql
+VACUUM mytable;
+```
+
+The `VACUUM` command collects table-level statistics such as the number of rows and pages. Vacuum all tables after loading data.
+
+### Configuring the Free Space Map<a id="topic10"></a>
+
+Expired rows are held in the *free space map*. The free space map must be sized large enough to hold all expired rows in your database. If not, a regular `VACUUM` command cannot reclaim space occupied by expired rows that overflow the free space map.
+
+**Note:** `VACUUM FULL` is not recommended with HAWQ because it is not safe for large tables and may take an unacceptably long time to complete. See [VACUUM](../reference/sql/VACUUM.html#topic1).
+
+Size the free space map with the following server configuration parameters:
+
+- `max_fsm_pages`
+- `max_fsm_relations`
diff --git a/datamgmt/ConcurrencyControl.html.md.erb b/datamgmt/ConcurrencyControl.html.md.erb
new file mode 100644
index 0000000..0291ddb
--- /dev/null
+++ b/datamgmt/ConcurrencyControl.html.md.erb
@@ -0,0 +1,31 @@
+---
+title: Concurrency Control
+---
+
+This topic discusses the mechanisms used in HAWQ to provide concurrency control.
+
+HAWQ and PostgreSQL do not use locks for concurrency control. They maintain data consistency using a multiversion model, Multiversion Concurrency Control (MVCC). MVCC achieves transaction isolation for each database session, and each query transaction sees a snapshot of data. This ensures the transaction sees consistent data that is not affected by other concurrent transactions.
+
+Because MVCC does not use explicit locks for concurrency control, lock contention is minimized and HAWQ maintains reasonable performance in multiuser environments. Locks acquired for querying (reading) data do not conflict with locks acquired for writing data.
+
+HAWQ provides multiple lock modes to control concurrent access to data in tables. Most HAWQ SQL commands automatically acquire the appropriate locks to ensure that referenced tables are not dropped or modified in incompatible ways while a command executes. For applications that cannot adapt easily to MVCC behavior, you can use the `LOCK` command to acquire explicit locks. However, proper use of MVCC generally provides better performance.
+
+<caption><span class="tablecap">Table 1. Lock Modes in HAWQ</span></caption>
+
+<a id="topic_f5l_qnh_kr__ix140861"></a>
+
+| Lock Mode | Associated SQL Commands | Conflicts With |
+|------------------------|-------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------|
+| ACCESS SHARE | `SELECT` | ACCESS EXCLUSIVE |
+| ROW SHARE | `SELECT FOR UPDATE`, `SELECT FOR SHARE` | EXCLUSIVE, ACCESS EXCLUSIVE |
+| ROW EXCLUSIVE | `INSERT`, `COPY` | SHARE, SHARE ROW EXCLUSIVE, EXCLUSIVE, ACCESS EXCLUSIVE |
+| SHARE UPDATE EXCLUSIVE | `VACUUM` (without `FULL`), `ANALYZE` | SHARE UPDATE EXCLUSIVE, SHARE, SHARE ROW EXCLUSIVE, EXCLUSIVE, ACCESS EXCLUSIVE |
+| SHARE | `CREATE INDEX` | ROW EXCLUSIVE, SHARE UPDATE EXCLUSIVE, SHARE ROW EXCLUSIVE, EXCLUSIVE, ACCESS EXCLUSIVE |
+| SHARE ROW EXCLUSIVE | | ROW EXCLUSIVE, SHARE UPDATE EXCLUSIVE, SHARE, SHARE ROW EXCLUSIVE, EXCLUSIVE, ACCESS EXCLUSIVE |
+| EXCLUSIVE | `DELETE`, `UPDATE` See [Note](#topic_f5l_qnh_kr__lock_note) | ROW SHARE, ROW EXCLUSIVE, SHARE UPDATE EXCLUSIVE, SHARE, SHARE ROW EXCLUSIVE, EXCLUSIVE, ACCESS EXCLUSIVE |
+| ACCESS EXCLUSIVE | `ALTER TABLE`, `DROP TABLE`, `TRUNCATE`, `REINDEX`, `CLUSTER`, `VACUUM FULL` | ACCESS SHARE, ROW SHARE, ROW EXCLUSIVE, SHARE UPDATE EXCLUSIVE, SHARE, SHARE ROW EXCLUSIVE, EXCLUSIVE, ACCESS EXCLUSIVE |
+
+
+**Note:** In HAWQ, `UPDATE` and `DELETE` acquire the more restrictive lock EXCLUSIVE rather than ROW EXCLUSIVE.
+
+
diff --git a/datamgmt/HAWQInputFormatforMapReduce.html.md.erb b/datamgmt/HAWQInputFormatforMapReduce.html.md.erb
new file mode 100644
index 0000000..9061014
--- /dev/null
+++ b/datamgmt/HAWQInputFormatforMapReduce.html.md.erb
@@ -0,0 +1,266 @@
+---
+title: HAWQ InputFormat for MapReduce
+---
+
+MapReduce is a programming model developed by Google for processing and generating large data sets on an array of commodity servers. You can use the HAWQ InputFormat option to enable MapReduce jobs to access HAWQ data stored in HDFS.
+
+To use HAWQ InputFormat, you only need to provide the URL of the databasen to connect to, along with the table name you want to access. HAWQ InputFormat only fetches the metadata of the database and table of interest, which is much less data than the table data itself. After getting the metadata, the HAWQ InputFormat determines where and how the table data is stored in HDFS. It reads and parses those HDFS files and processes the parsed table tuples directly inside a Map task.
+
+This chapter describes the document format and schema for defining HAWQ MapReduce jobs.
+
+## Supported Data Types<a id="supporteddatatypes"></a>
+
+HAWQ InputFormat supports the following data types:
+
+| SQL/HAWQ | JDBC/JAVA | setXXX | getXXX |
+|-------------------------|--------------------------------------------------|---------------|---------------|
+| DECIMAL/NUMERIC | java.math.BigDecimal | setBigDecimal | getBigDecimal |
+| FLOAT8/DOUBLE PRECISION | double | setDouble | getDouble |
+| INT8/BIGINT | long | setLong | getLong |
+| INTEGER/INT4/INT | int | setInt | getInt |
+| FLOAT4/REAL | float | setFloat | getFloat |
+| SMALLINT/INT2 | short | setShort | getShort |
+| BOOL/BOOLEAN | boolean | setBoolean | getBoolean |
+| VARCHAR/CHAR/TEXT | String | setString | getString |
+| DATE | java.sql.Date | setDate | getDate |
+| TIME/TIMETZ | java.sql.Time | setTime | getTime |
+| TIMESTAMP/TIMSTAMPTZ | java.sql.Timestamp | setTimestamp | getTimestamp |
+| ARRAY | java.sq.Array | setArray | getArray |
+| BIT/VARBIT | com.pivotal.hawq.mapreduce.datatype. | setVarbit | getVarbit |
+| BYTEA | byte\[\] | setByte | getByte |
+| INTERVAL | com.pivotal.hawq.mapreduce.datatype.HAWQInterval | setInterval | getInterval |
+| POINT | com.pivotal.hawq.mapreduce.datatype.HAWQPoint | setPoint | getPoint |
+| LSEG | com.pivotal.hawq.mapreduce.datatype.HAWQLseg | setLseg | getLseg |
+| BOX | com.pivotal.hawq.mapreduce.datatype.HAWQBox | setBox | getBox |
+| CIRCLE | com.pivotal.hawq.mapreduce.datatype.HAWQCircle | setVircle | getCircle |
+| PATH | com.pivotal.hawq.mapreduce.datatype.HAWQPath | setPath | getPath |
+| POLYGON | com.pivotal.hawq.mapreduce.datatype.HAWQPolygon | setPolygon | getPolygon |
+| MACADDR | com.pivotal.hawq.mapreduce.datatype.HAWQMacaddr | setMacaddr | getMacaddr |
+| INET | com.pivotal.hawq.mapreduce.datatype.HAWQInet | setInet | getInet |
+| CIDR | com.pivotal.hawq.mapreduce.datatype.HAWQCIDR | setCIDR | getCIDR |
+
+## HAWQ InputFormat Example<a id="hawqinputformatexample"></a>
+
+The following example shows how you can use the HAWQ InputFormat to access HAWQ table data from MapReduce jobs.
+
+``` java
+package com.mycompany.app;
+import com.pivotal.hawq.mapreduce.HAWQException;
+import com.pivotal.hawq.mapreduce.HAWQInputFormat;
+import com.pivotal.hawq.mapreduce.HAWQRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.conf.Configured;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.mapreduce.Job;
+import org.apache.hadoop.mapreduce.Mapper;
+import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
+import org.apache.hadoop.util.Tool;
+import org.apache.hadoop.util.ToolRunner;
+import org.apache.hadoop.io.IntWritable;
+
+import java.io.IOException;
+public class HAWQInputFormatDemoDriver extends Configured
+implements Tool {
+
+ // CREATE TABLE employees (
+ // id INTEGER NOT NULL, name VARCHAR(32) NOT NULL);
+ public static class DemoMapper extends
+ Mapper<Void, HAWQRecord, IntWritable, Text> {
+ int id = 0;
+ String name = null;
+ public void map(Void key, HAWQRecord value, Context context)
+ throws IOException, InterruptedException {
+ try {
+ id = value.getInt(1);
+ name = value.getString(2);
+ } catch (HAWQException hawqE) {
+ throw new IOException(hawqE.getMessage());
+ }
+ context.write(new IntWritable(id), new Text(name));
+ }
+ }
+ private static int printUsage() {
+ System.out.println("HAWQInputFormatDemoDriver
+ <database_url> <table_name> <output_path> [username]
+ [password]");
+ ToolRunner.printGenericCommandUsage(System.out);
+ return 2;
+ }
+
+ public int run(String[] args) throws Exception {
+ if (args.length < 3) {
+ return printUsage();
+ }
+ Job job = new Job(getConf());
+ job.setJobName("hawq-inputformat-demo");
+ job.setJarByClass(HAWQInputFormatDemoDriver.class);
+ job.setMapperClass(DemoMapper.class);
+ job.setMapOutputValueClass(Text.class);
+ job.setOutputValueClass(Text.class);
+ String db_url = args[0];
+ String table_name = args[1];
+ String output_path = args[2];
+ String user_name = null;
+ if (args.length > 3) {
+ user_name = args[3];
+ }
+ String password = null;
+ if (args.length > 4) {
+ password = args[4];
+ }
+ job.setInputFormatClass(HAWQInputFormat.class);
+ HAWQInputFormat.setInput(job.getConfiguration(), db_url,
+ user_name, password, table_name);
+ FileOutputFormat.setOutputPath(job, new
+ Path(output_path));
+ job.setNumReduceTasks(0);
+ int res = job.waitForCompletion(true) ? 0 : 1;
+ return res;
+ }
+
+ public static void main(String[] args) throws Exception {
+ int res = ToolRunner.run(new Configuration(),
+ new HAWQInputFormatDemoDriver(), args);
+ System.exit(res);
+ }
+}
+```
+
+**To compile and run the example:**
+
+1. Add the following dependencies into the project for compilation:
+ 1. HAWQInputFormat jars (located in the `$GPHOME/lib/postgresql/hawq-mr-io` directory):
+ - `hawq-mapreduce-common.jar`
+ - `hawq-mapreduce-ao.jar`
+ - `hawq_mapreduce-parquet.jar`
+ - `hawq-mapreduce-tool.jar`
+
+ 2. Required 3rd party jars (located in the `$GPHOME/lib/postgresql/hawq-mr-io/lib` directory):
+ - `parquet-column-1.1.0.jar`
+ - `parquet-common-1.1.0.jar`
+ - `parquet-encoding-1.1.0.jar`
+ - `parquet-format-1.1.0.jar`
+ - `parquet-hadoop-1.1.0.jar`
+ - `postgresql-jdbc.jar`
+ - `snakeyaml.jar`
+
+ 3. Hadoop Mapreduce related jars (located in the install directory of your Hadoop distribution).
+
+2. Check that you have installed HAWQ, HDFS and Yarn.
+3. Create sample table:
+ 1. Log in to HAWQ:
+
+ ``` shell
+ $ psql -d postgres
+ ```
+
+ 2. Create the sample table:
+
+ ``` sql
+ CREATE TABLE employees (
+ id INTEGER NOT NULL,
+ name TEXT NOT NULL);
+ ```
+
+ Or a Parquet table:
+
+ ``` sql
+ CREATE TABLE employees ( id INTEGER NOT NULL, name TEXT NOT NULL) with (appendonly=true, orientation=parquet);
+ ```
+
+ 3. Insert one tuple:
+
+ ``` sql
+ INSERT INTO employees VALUES (1, 'Paul');
+ ```
+
+ 4. Use the following shell script snippet showing how to run the Mapreduce job:
+
+ ``` shell
+ # Suppose all five needed jars are under ./lib
+ export
+ LIBJARS=lib/hawq-mapreduce-common.jar,lib/hawq-mapreduce-ao.
+ jar,lib/hawq-mapreduce-tool.jar,lib/postgresql-9.2-1003-jdbc
+ 4.jar,lib/snakeyaml-1.12.jar
+ export
+ HADOOP_CLASSPATH=lib/hawq-mapreduce-common.jar:lib/hawq-mapr
+ educe-ao.jar:lib/hawq-mapreduce-tool.jar:lib/postgresql-9.2-
+ 1003-jdbc4.jar:lib/snakeyaml-1.12.jar
+ # Suppose the built application jar is my-app.jar
+ hadoop jar my-app.jar
+ com.mycompany.app.HAWQInputFormatDemoDriver -libjars
+ ${LIBJARS} localhost:5432/postgres employees /tmp/employees
+ ```
+
+ 5. Use the following command to check the result of the Mapreduce job:
+
+ ``` shell
+ $ hadoop fs -cat /tmp/employees/*
+ ```
+
+The output will appear as follows:
+
+``` pre
+1 Paul
+```
+
+## Accessing HAWQ Data<a id="accessinghawqdata"></a>
+
+You can access HAWQ data using the following interfaces:
+
+- HAWQInputFormat.setInput API: Use this when HAWQ is running.
+- Metadata Export Tool: Use this when HAWQ is not running.
+
+### HAWQInputFormat.setInput<a id="hawqinputformatsetinput"></a>
+
+``` java
+ /**
+ * Initializes the map-part of the job with the appropriate input settings
+ * through connecting to Database.
+ *
+ * @param conf
+ * The map-reduce job configuration
+ * @param db_url
+ * The database URL to connect to
+ * @param username
+ * The username for setting up a connection to the database
+ * @param password
+ * The password for setting up a connection to the database
+ * @param tableName
+ * The name of the table to access to
+ * @throws Exception
+ */
+public static void setInput(Configuration conf, String db_url,
+ String username, String password, String tableName)
+throws Exception;
+```
+
+### Metadata Export Tool<a id="metadataexporttool"></a>
+
+Use the metadata export tool, `hawq extract`, to export the metadata of the target table into a local YAML file:
+
+``` shell
+$ hawq extract [-h hostname] [-p port] [-U username] [-ddatabase] [-o output_file] [-W] <tablename>
+```
+
+Using the extracted metadata, access HAWQ data through the following interface:
+
+``` java
+ /**
+ * Initializes the map-part of the job with the appropriate input settings through reading metadata file stored in local filesystem.
+ *
+ * To get metadata file, please use hawq extract first
+ *
+ * @param conf
+ * The map-reduce job configuration
+ * @param pathStr
+ * The metadata file path in local filesystem. e.g.
+ * /home/gpadmin/metadata/postgres_test
+ * @throws Exception
+ */
+public static void setInput(Configuration conf, String pathStr)
+ throws Exception;
+```
+
+
diff --git a/datamgmt/Transactions.html.md.erb b/datamgmt/Transactions.html.md.erb
new file mode 100644
index 0000000..d7cf6ee
--- /dev/null
+++ b/datamgmt/Transactions.html.md.erb
@@ -0,0 +1,58 @@
+---
+title: Working with Transactions
+---
+
+This topic describes transaction support in HAWQ.
+
+Transactions allow you to bundle multiple SQL statements in one all-or-nothing operation.
+
+The following are the HAWQ SQL transaction commands:
+
+- `BEGIN` or `START TRANSACTION `starts a transaction block.
+- `END` or `COMMIT` commits the results of a transaction.
+- `ROLLBACK` abandons a transaction without making any changes.
+- `SAVEPOINT` marks a place in a transaction and enables partial rollback. You can roll back commands executed after a savepoint while maintaining commands executed before the savepoint.
+- `ROLLBACK TO SAVEPOINT `rolls back a transaction to a savepoint.
+- `RELEASE SAVEPOINT `destroys a savepoint within a transaction.
+
+## Transaction Isolation Levels<a id="topic8"></a>
+
+HAWQ accepts the standard SQL transaction levels as follows:
+
+- *read uncommitted* and *read committed* behave like the standard *read committed*
+- serializable and repeatable read behave like the standard serializable
+
+The following information describes the behavior of the HAWQ transaction levels:
+
+- **read committed/read uncommitted** — Provides fast, simple, partial transaction isolation. With read committed and read uncommitted transaction isolation, `SELECT`, `UPDATE`, and `DELETE` transactions operate on a snapshot of the database taken when the query started.
+
+A `SELECT` query:
+
+- Sees data committed before the query starts.
+- Sees updates executed within the transaction.
+- Does not see uncommitted data outside the transaction.
+- Can possibly see changes that concurrent transactions made if the concurrent transaction is committed after the initial read in its own transaction.
+
+Successive `SELECT` queries in the same transaction can see different data if other concurrent transactions commit changes before the queries start. `UPDATE` and `DELETE` commands find only rows committed before the commands started.
+
+Read committed or read uncommitted transaction isolation allows concurrent transactions to modify or lock a row before `UPDATE` or `DELETE` finds the row. Read committed or read uncommitted transaction isolation may be inadequate for applications that perform complex queries and updates and require a consistent view of the database.
+
+- **serializable/repeatable read** — Provides strict transaction isolation in which transactions execute as if they run one after another rather than concurrently. Applications on the serializable or repeatable read level must be designed to retry transactions in case of serialization failures.
+
+A `SELECT` query:
+
+- Sees a snapshot of the data as of the start of the transaction (not as of the start of the current query within the transaction).
+- Sees only data committed before the query starts.
+- Sees updates executed within the transaction.
+- Does not see uncommitted data outside the transaction.
+- Does not see changes that concurrent transactions made.
+
+ Successive `SELECT` commands within a single transaction always see the same data.
+
+ `UPDATE`, `DELETE, SELECT FOR UPDATE,` and `SELECT FOR SHARE` commands find only rows committed before the command started. If a concurrent transaction has already updated, deleted, or locked a target row when the row is found, the serializable or repeatable read transaction waits for the concurrent transaction to update the row, delete the row, or roll back.
+
+ If the concurrent transaction updates or deletes the row, the serializable or repeatable read transaction rolls back. If the concurrent transaction rolls back, then the serializable or repeatable read transaction updates or deletes the row.
+
+The default transaction isolation level in HAWQ is *read committed*. To change the isolation level for a transaction, declare the isolation level when you `BEGIN` the transaction or use the `SET TRANSACTION` command after the transaction starts.
+
+
diff --git a/datamgmt/about_statistics.html.md.erb b/datamgmt/about_statistics.html.md.erb
new file mode 100644
index 0000000..af21251
--- /dev/null
+++ b/datamgmt/about_statistics.html.md.erb
@@ -0,0 +1,187 @@
+---
+title: About Database Statistics
+---
+
+## Overview<a id="overview"></a>
+
+Statistics are metadata that describe the data stored in the database. The query optimizer needs up-to-date statistics to choose the best execution plan for a query. For example, if a query joins two tables and one of them must be broadcast to all segments, the optimizer can choose the smaller of the two tables to minimize network traffic.
+
+The statistics used by the optimizer are calculated and saved in the system catalog by the `ANALYZE` command. There are three ways to initiate an analyze operation:
+
+- You can run the `ANALYZE` command directly.
+- You can run the `analyzedb` management utility outside of the database, at the command line.
+- An automatic analyze operation can be triggered when DML operations are performed on tables that have no statistics or when a DML operation modifies a number of rows greater than a specified threshold.
+
+These methods are described in the following sections.
+
+Calculating statistics consumes time and resources, so HAWQ produces estimates by calculating statistics on samples of large tables. In most cases, the default settings provide the information needed to generate correct execution plans for queries. If the statistics produced are not producing optimal query execution plans, the administrator can tune configuration parameters to produce more accurate stastistics by increasing the sample size or the granularity of statistics saved in the system catalog. Producing more accurate statistics has CPU and storage costs and may not produce better plans, so it is important to view explain plans and test query performance to ensure that the additional statistics-related costs result in better query performance.
+
+## System Statistics<a id="topic_oq3_qxj_3s"></a>
+
+### Table Size<a id="tablesize"></a>
+
+The query planner seeks to minimize the disk I/O and network traffic required to execute a query, using estimates of the number of rows that must be processed and the number of disk pages the query must access. The data from which these estimates are derived are the `pg_class` system table columns `reltuples` and `relpages`, which contain the number of rows and pages at the time a `VACUUM` or `ANALYZE` command was last run. As rows are added or deleted, the numbers become less accurate. However, an accurate count of disk pages is always available from the operating system, so as long as the ratio of `reltuples` to `relpages` does not change significantly, the optimizer can produce an estimate of the number of rows that is sufficiently accurate to choose the correct query execution plan.
+
+In append-optimized tables, the number of tuples is kept up-to-date in the system catalogs, so the `reltuples` statistic is not an estimate. Non-visible tuples in the table are subtracted from the total. The `relpages` value is estimated from the append-optimized block sizes.
+
+When the `reltuples` column differs significantly from the row count returned by `SELECT COUNT(*)`, an analyze should be performed to update the statistics.
+
+### The pg\_statistic System Table and pg\_stats View<a id="views"></a>
+
+The `pg_statistic` system table holds the results of the last `ANALYZE` operation on each database table. There is a row for each column of every table. It has the following columns:
+
+starelid
+The object ID of the table or index the column belongs to.
+
+staatnum
+The number of the described column, beginning with 1.
+
+stanullfrac
+The fraction of the column's entries that are null.
+
+stawidth
+The average stored width, in bytes, of non-null entries.
+
+stadistinct
+The number of distinct nonnull data values in the column.
+
+stakind*N*
+A code number indicating the kind of statistics stored in the *N*th slot of the `pg_statistic` row.
+
+staop*N*
+An operator used to derive the statistics stored in the *N*th slot.
+
+stanumbers*N*
+Numerical statistics of the appropriate kind for the *N*th slot, or NULL if the slot kind does not involve numerical values.
+
+stavalues*N*
+Column data values of the appropriate kind for the *N*th slot, or NULL if the slot kind does not store any data values.
+
+The statistics collected for a column vary for different data types, so the `pg_statistic` table stores statistics that are appropriate for the data type in four *slots*, consisting of four columns per slot. For example, the first slot, which normally contains the most common values for a column, consists of the columns `stakind1`, `staop1`, `stanumbers1`, and `stavalues1`. Also see [pg\_statistic](../reference/catalog/pg_statistic.html#topic1).
+
+The `stakindN` columns each contain a numeric code to describe the type of statistics stored in their slot. The `stakind` code numbers from 1 to 99 are reserved for core PostgreSQL data types. HAWQ uses code numbers 1, 2, and 3. A value of 0 means the slot is unused. The following table describes the kinds of statistics stored for the three codes.
+
+<a id="topic_oq3_qxj_3s__table_upf_1yc_nt"></a>
+
+<table>
+<caption><span class="tablecap">Table 1. Contents of pg_statistic "slots"</span></caption>
+<colgroup>
+<col width="50%" />
+<col width="50%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>stakind Code</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>1</td>
+<td><em>Most CommonValues (MCV) Slot</em>
+<ul>
+<li><code class="ph codeph">staop</code> contains the object ID of the "=" operator, used to decide whether values are the same or not.</li>
+<li><code class="ph codeph">stavalues</code> contains an array of the <em>K</em> most common non-null values appearing in the column.</li>
+<li><code class="ph codeph">stanumbers</code> contains the frequencies (fractions of total row count) of the values in the <code class="ph codeph">stavalues</code> array.</li>
+</ul>
+The values are ordered in decreasing frequency. Since the arrays are variable-size, <em>K</em> can be chosen by the statistics collector. Values must occur more than once to be added to the <code class="ph codeph">stavalues</code> array; a unique column has no MCV slot.</td>
+</tr>
+<tr class="even">
+<td>2</td>
+<td><em>Histogram Slot</em> – describes the distribution of scalar data.
+<ul>
+<li><code class="ph codeph">staop</code> is the object ID of the "<" operator, which describes the sort ordering.</li>
+<li><code class="ph codeph">stavalues</code> contains <em>M</em> (where <em>M</em>>=2) non-null values that divide the non-null column data values into <em>M</em>-1 bins of approximately equal population. The first <code class="ph codeph">stavalues</code> item is the minimum value and the last is the maximum value.</li>
+<li><code class="ph codeph">stanumbers</code> is not used and should be null.</li>
+</ul>
+<p>If a Most Common Values slot is also provided, then the histogram describes the data distribution after removing the values listed in the MCV array. (It is a <em>compressed histogram</em> in the technical parlance). This allows a more accurate representation of the distribution of a column with some very common values. In a column with only a few distinct values, it is possible that the MCV list describes the entire data population; in this case the histogram reduces to empty and should be omitted.</p></td>
+</tr>
+<tr class="odd">
+<td>3</td>
+<td><em>Correlation Slot</em> – describes the correlation between the physical order of table tuples and the ordering of data values of this column.
+<ul>
+<li><code class="ph codeph">staop</code> is the object ID of the "<" operator. As with the histogram, more than one entry could theoretically appear.</li>
+<li><code class="ph codeph">stavalues</code> is not used and should be NULL.</li>
+<li><code class="ph codeph">stanumbers</code> contains a single entry, the correlation coefficient between the sequence of data values and the sequence of their actual tuple positions. The coefficient ranges from +1 to -1.</li>
+</ul></td>
+</tr>
+</tbody>
+</table>
+
+The `pg_stats` view presents the contents of `pg_statistic` in a friendlier format. For more information, see [pg\_stats](../reference/catalog/pg_stats.html#topic1).
+
+Newly created tables and indexes have no statistics.
+
+### Sampling<a id="topic_oq3_qxj_3s__section_wsy_1rv_mt"></a>
+
+When calculating statistics for large tables, HAWQ creates a smaller table by sampling the base table. If the table is partitioned, samples are taken from all partitions.
+
+If a sample table is created, the number of rows in the sample is calculated to provide a maximum acceptable relative error. The amount of acceptable error is specified with the `gp_analyze_relative_error` system configuration parameter, which is set to .25 (25%) by default. This is usually sufficiently accurate to generate correct query plans. If `ANALYZE` is not producing good estimates for a table column, you can increase the sample size by setting the `gp_analyze_relative_error` configuration parameter to a lower value. Beware that setting this parameter to a low value can lead to a very large sample size and dramatically increase analyze time.
+
+### Updating Statistics<a id="topic_oq3_qxj_3s__section_u5p_brv_mt"></a>
+
+Running `ANALYZE` with no arguments updates statistics for all tables in the database. This could take a very long time, so it is better to analyze tables selectively after data has changed. You can also analyze a subset of the columns in a table, for example columns used in joins, `WHERE` clauses, `SORT` clauses, `GROUP BY` clauses, or `HAVING` clauses.
+
+Analyzing a severely bloated table can generate poor statistics if the sample contains empty pages, so it is good practice to vacuum a bloated table before analyzing it.
+
+See the SQL Command Reference for details of running the `ANALYZE` command.
+
+Refer to the Management Utility Reference for details of running the `analyzedb` command.
+
+### Analyzing Partitioned and Append-Optimized Tables<a id="topic_oq3_qxj_3s__section_cv2_crv_mt"></a>
+
+When the `ANALYZE` command is run on a partitioned table, it analyzes each leaf-level subpartition, one at a time. You can run `ANALYZE` on just new or changed partition files to avoid analyzing partitions that have not changed. If a table is partitioned, you can analyze just new or changed partitions.
+
+The `analyzedb` command-line utility skips unchanged partitions automatically. It also runs concurrent sessions so it can analyze several partitions concurrently. It runs five sessions by default, but the number of sessions can be set from 1 to 10 with the `-p` command-line option. Each time `analyzedb` runs, it saves state information for append-optimized tables and partitions in the `db_analyze` directory in the master data directory. The next time it runs, `analyzedb` compares the current state of each table with the saved state and skips analyzing a table or partition if it is unchanged. Heap tables are always analyzed.
+
+If the Pivotal Query Optimizer is enabled, you also need to run `ANALYZE ROOTPARTITION` to refresh the root partition statistics. The Pivotal Query Optimizer requires statistics at the root level for partitioned tables. The legacy optimizer does not use these statistics. Enable the Pivotal Query Optimizer by setting both the `optimizer` and `optimizer_analyze_root_partition` system configuration parameters to on. The root level statistics are then updated when you run `ANALYZE` or `ANALYZE ROOTPARTITION`. The time to run `ANALYZE ROOTPARTITION` is similar to the time to analyze a single partition since `ANALYZE ROOTPARTITION`. The `analyzedb` utility updates root partition statistics by default .
+
+## Configuring Statistics<a id="topic_gyb_qrd_2t"></a>
+
+There are several options for configuring HAWQ statistics collection.
+
+### Statistics Target<a id="statstarget"></a>
+
+The statistics target is the size of the `most_common_vals`, `most_common_freqs`, and `histogram_bounds` arrays for an individual column. By default, the target is 25. The default target can be changed by setting a server configuration parameter and the target can be set for any column using the `ALTER TABLE` command. Larger values increase the time needed to do `ANALYZE`, but may improve the quality of the legacy query optimizer (planner) estimates.
+
+Set the system default statistics target to a different value by setting the `default_statistics_target` server configuration parameter. The default value is usually sufficient, and you should only raise or lower it if your tests demonstrate that query plans improve with the new target. For example, to raise the default statistics target from 25 to 50 you can use the `hawq config` utility:
+
+``` shell
+$ hawq config -c default_statistics_target -v 50
+```
+
+The statististics target for individual columns can be set with the `ALTER TABLE` command. For example, some queries can be improved by increasing the target for certain columns, especially columns that have irregular distributions. You can set the target to zero for columns that never contribute to query otpimization. When the target is 0, `ANALYZE` ignores the column. For example, the following `ALTER TABLE` command sets the statistics target for the `notes` column in the `emp` table to zero:
+
+``` sql
+ALTER TABLE emp ALTER COLUMN notes SET STATISTICS 0;
+```
+
+The statistics target can be set in the range 0 to 1000, or set it to -1 to revert to using the system default statistics target.
+
+Setting the statistics target on a parent partition table affects the child partitions. If you set statistics to 0 on some columns on the parent table, the statistics for the same columns are set to 0 for all children partitions. However, if you later add or exchange another child partition, the new child partition will use either the default statistics target or, in the case of an exchange, the previous statistics target. Therefore, if you add or exchange child partitions, you should set the statistics targets on the new child table.
+
+### Automatic Statistics Collection<a id="topic_gyb_qrd_2t__section_j3p_drv_mt"></a>
+
+HAWQ can be set to automatically run `ANALYZE` on a table that either has no statistics or has changed significantly when certain operations are performed on the table. For partitioned tables, automatic statistics collection is only triggered when the operation is run directly on a leaf table, and then only the leaf table is analyzed.
+
+Automatic statistics collection has three modes:
+
+- `none` disables automatic statistics collection.
+- `on_no_stats` triggers an analyze operation for a table with no existing statistics when any of the commands `CREATE TABLE AS SELECT`, `INSERT`, or `COPY` are executed on the table.
+- `on_change` triggers an analyze operation when any of the commands `CREATE TABLE AS SELECT`, `UPDATE`, `DELETE`, `INSERT`, or `COPY` are executed on the table and the number of rows affected exceeds the threshold defined by the `gp_autostats_on_change_threshold` configuration parameter.
+
+The automatic statistics collection mode is set separately for commands that occur within a procedural language function and commands that execute outside of a function:
+
+- The `gp_autostats_mode` configuration parameter controls automatic statistics collection behavior outside of functions and is set to `on_no_stats` by default.
+
+With the `on_change` mode, `ANALYZE` is triggered only if the number of rows affected exceeds the threshold defined by the `gp_autostats_on_change_threshold` configuration parameter. The default value for this parameter is a very high value, 2147483647, which effectively disables automatic statistics collection; you must set the threshold to a lower number to enable it. The `on_change` mode could trigger large, unexpected analyze operations that could disrupt the system, so it is not recommended to set it globally. It could be useful in a session, for example to automatically analyze a table following a load.
+
+To disable automatic statistics collection outside of functions, set the `gp_autostats_mode` parameter to `none`:
+
+``` shell
+$ hawq configure -c gp_autostats_mode -v none
+```
+
+Set the `log_autostats` system configuration parameter to on if you want to log automatic statistics collection operations.
+
+
diff --git a/datamgmt/dml.html.md.erb b/datamgmt/dml.html.md.erb
new file mode 100644
index 0000000..9a15d78
--- /dev/null
+++ b/datamgmt/dml.html.md.erb
@@ -0,0 +1,35 @@
+---
+title: Managing Data
+---
+
+This chapter provides information about manipulating data and concurrent access in HAWQ.
+
+- **[Basic Data Operations](../datamgmt/BasicDataOperations.html)**
+
+ This topic describes basic data operations that you perform in HAWQ.
+
+- **[About Database Statistics](../datamgmt/about_statistics.html)**
+
+ An overview of statistics gathered by the `ANALYZE` command in HAWQ.
+
+- **[Concurrency Control](../datamgmt/ConcurrencyControl.html)**
+
+ This topic discusses the mechanisms used in HAWQ to provide concurrency control.
+
+- **[Working with Transactions](../datamgmt/Transactions.html)**
+
+ This topic describes transaction support in HAWQ.
+
+- **[Loading and Unloading Data](../datamgmt/load/g-loading-and-unloading-data.html)**
+
+ The topics in this section describe methods for loading and writing data into and out of HAWQ, and how to format data files.
+
+- **[Working with PXF and External Data](../pxf/PivotalExtensionFrameworkPXF.html)**
+
+ HAWQ Extension Framework (PXF) is an extensible framework that allows HAWQ to query external system data.
+
+- **[HAWQ InputFormat for MapReduce](../datamgmt/HAWQInputFormatforMapReduce.html)**
+
+ MapReduce is a programming model developed by Google for processing and generating large data sets on an array of commodity servers. You can use the HAWQ InputFormat option to enable MapReduce jobs to access HAWQ data stored in HDFS.
+
+
diff --git a/datamgmt/load/creating-external-tables-examples.html.md.erb b/datamgmt/load/creating-external-tables-examples.html.md.erb
new file mode 100644
index 0000000..aaa82d8
--- /dev/null
+++ b/datamgmt/load/creating-external-tables-examples.html.md.erb
@@ -0,0 +1,117 @@
+---
+title: Creating External Tables - Examples
+---
+
+The following examples show how to define external data with different protocols. Each `CREATE EXTERNAL TABLE` command can contain only one protocol.
+
+**Note:** When using IPv6, always enclose the numeric IP addresses in square brackets.
+
+Start `gpfdist` before you create external tables with the `gpfdist` protocol. The following code starts the `gpfdist` file server program in the background on port *8081* serving files from directory `/var/data/staging`. The logs are saved in `/home/gpadmin/log`.
+
+``` shell
+$ gpfdist -p 8081 -d /var/data/staging -l /home/gpadmin/log &
+```
+
+## Example 1 - Single gpfdist instance on single-NIC machine<a id="ex1"></a>
+
+Creates a readable external table, `ext_expenses`, using the `gpfdist` protocol. The files are formatted with a pipe (|) as the column delimiter.
+
+``` sql
+=# CREATE EXTERNAL TABLE ext_expenses
+ ( name text, date date, amount float4, category text, desc1 text )
+ LOCATION ('gpfdist://etlhost-1:8081/*', 'gpfdist://etlhost-1:8082/*')
+ FORMAT 'TEXT' (DELIMITER '|');
+```
+
+## Example 2 - Multiple gpfdist instances<a id="ex2"></a>
+
+Creates a readable external table, *ext\_expenses*, using the `gpfdist` protocol from all files with the *txt* extension. The column delimiter is a pipe ( | ) and NULL is a space (' ').
+
+``` sql
+=# CREATE EXTERNAL TABLE ext_expenses
+ ( name text, date date, amount float4, category text, desc1 text )
+ LOCATION ('gpfdist://etlhost-1:8081/*.txt', 'gpfdist://etlhost-2:8081/*.txt')
+ FORMAT 'TEXT' ( DELIMITER '|' NULL ' ') ;
+
+```
+
+## Example 3 - Multiple gpfdists instances<a id="ex3"></a>
+
+Creates a readable external table, *ext\_expenses,* from all files with the *txt* extension using the `gpfdists` protocol. The column delimiter is a pipe ( | ) and NULL is a space (' '). For information about the location of security certificates, see [gpfdists Protocol](g-gpfdists-protocol.html).
+
+1. Run `gpfdist` with the `--ssl` option.
+2. Run the following command.
+
+ ``` sql
+ =# CREATE EXTERNAL TABLE ext_expenses
+ ( name text, date date, amount float4, category text, desc1 text )
+ LOCATION ('gpfdists://etlhost-1:8081/*.txt', 'gpfdists://etlhost-2:8082/*.txt')
+ FORMAT 'TEXT' ( DELIMITER '|' NULL ' ') ;
+
+ ```
+
+## Example 4 - Single gpfdist instance with error logging<a id="ex4"></a>
+
+Uses the gpfdist protocol to create a readable external table, `ext_expenses,` from all files with the *txt* extension. The column delimiter is a pipe ( | ) and NULL (' ') is a space.
+
+Access to the external table is single row error isolation mode. Input data formatting errors can be captured so that you can view the errors, fix the issues, and then reload the rejected data. If the error count on a segment is greater than five (the `SEGMENT REJECT LIMIT` value), the entire external table operation fails and no rows are processed.
+
+``` sql
+=# CREATE EXTERNAL TABLE ext_expenses
+ ( name text, date date, amount float4, category text, desc1 text )
+ LOCATION ('gpfdist://etlhost-1:8081/*.txt', 'gpfdist://etlhost-2:8082/*.txt')
+ FORMAT 'TEXT' ( DELIMITER '|' NULL ' ')
+ LOG ERRORS INTO expenses_errs SEGMENT REJECT LIMIT 5;
+
+```
+
+To create the readable `ext_expenses` table from CSV-formatted text files:
+
+``` sql
+=# CREATE EXTERNAL TABLE ext_expenses
+ ( name text, date date, amount float4, category text, desc1 text )
+ LOCATION ('gpfdist://etlhost-1:8081/*.txt', 'gpfdist://etlhost-2:8082/*.txt')
+ FORMAT 'CSV' ( DELIMITER ',' )
+ LOG ERRORS INTO expenses_errs SEGMENT REJECT LIMIT 5;
+
+```
+
+## Example 5 - Readable Web External Table with Script<a id="ex5"></a>
+
+Creates a readable web external table that executes a script once on five virtual segments:
+
+``` sql
+=# CREATE EXTERNAL WEB TABLE log_output (linenum int, message text)
+ EXECUTE '/var/load_scripts/get_log_data.sh' ON 5
+ FORMAT 'TEXT' (DELIMITER '|');
+
+```
+
+## Example 6 - Writable External Table with gpfdist<a id="ex6"></a>
+
+Creates a writable external table, *sales\_out*, that uses `gpfdist` to write output data to the file *sales.out*. The column delimiter is a pipe ( | ) and NULL is a space (' '). The file will be created in the directory specified when you started the gpfdist file server.
+
+``` sql
+=# CREATE WRITABLE EXTERNAL TABLE sales_out (LIKE sales)
+ LOCATION ('gpfdist://etl1:8081/sales.out')
+ FORMAT 'TEXT' ( DELIMITER '|' NULL ' ')
+ DISTRIBUTED BY (txn_id);
+
+```
+
+## Example 7 - Writable External Web Table with Script<a id="ex7"></a>
+
+Creates a writable external web table, `campaign_out`, that pipes output data recieved by the segments to an executable script, `to_adreport_etl.sh`:
+
+``` sql
+=# CREATE WRITABLE EXTERNAL WEB TABLE campaign_out
+ (LIKE campaign)
+ EXECUTE '/var/unload_scripts/to_adreport_etl.sh' ON 6
+ FORMAT 'TEXT' (DELIMITER '|');
+```
+
+## Example 8 - Readable and Writable External Tables with XML Transformations<a id="ex8"></a>
+
+HAWQ can read and write XML data to and from external tables with gpfdist. For information about setting up an XML transform, see [Transforming XML Data](g-transforming-xml-data.html#topic75).
+
+
diff --git a/datamgmt/load/g-about-gpfdist-setup-and-performance.html.md.erb b/datamgmt/load/g-about-gpfdist-setup-and-performance.html.md.erb
new file mode 100644
index 0000000..28a0bfe
--- /dev/null
+++ b/datamgmt/load/g-about-gpfdist-setup-and-performance.html.md.erb
@@ -0,0 +1,22 @@
+---
+title: About gpfdist Setup and Performance
+---
+
+Consider the following scenarios for optimizing your ETL network performance.
+
+- Allow network traffic to use all ETL host Network Interface Cards (NICs) simultaneously. Run one instance of `gpfdist` on the ETL host, then declare the host name of each NIC in the `LOCATION` clause of your external table definition (see [Creating External Tables - Examples](creating-external-tables-examples.html#topic44)).
+
+<a id="topic14__du165872"></a>
+<span class="figtitleprefix">Figure: </span>External Table Using Single gpfdist Instance with Multiple NICs
+
+<img src="../../images/ext_tables_multinic.jpg" class="image" width="472" height="271" />
+
+- Divide external table data equally among multiple `gpfdist` instances on the ETL host. For example, on an ETL system with two NICs, run two `gpfdist` instances (one on each NIC) to optimize data load performance and divide the external table data files evenly between the two `gpfdists`.
+
+<a id="topic14__du165882"></a>
+
+<span class="figtitleprefix">Figure: </span>External Tables Using Multiple gpfdist Instances with Multiple NICs
+
+<img src="../../images/ext_tables.jpg" class="image" width="467" height="282" />
+
+**Note:** Use pipes (|) to separate formatted text when you submit files to `gpfdist`. HAWQ encloses comma-separated text strings in single or double quotes. `gpfdist` has to remove the quotes to parse the strings. Using pipes to separate formatted text avoids the extra step and improves performance.
diff --git a/datamgmt/load/g-character-encoding.html.md.erb b/datamgmt/load/g-character-encoding.html.md.erb
new file mode 100644
index 0000000..9f3756d
--- /dev/null
+++ b/datamgmt/load/g-character-encoding.html.md.erb
@@ -0,0 +1,11 @@
+---
+title: Character Encoding
+---
+
+Character encoding systems consist of a code that pairs each character from a character set with something else, such as a sequence of numbers or octets, to facilitate data stransmission and storage. HAWQ supports a variety of character sets, including single-byte character sets such as the ISO 8859 series and multiple-byte character sets such as EUC (Extended UNIX Code), UTF-8, and Mule internal code. Clients can use all supported character sets transparently, but a few are not supported for use within the server as a server-side encoding.
+
+Data files must be in a character encoding recognized by HAWQ. Data files that contain invalid or unsupported encoding sequences encounter errors when loading into HAWQ.
+
+**Note:** On data files generated on a Microsoft Windows operating system, run the `dos2unix` system command to remove any Windows-only characters before loading into HAWQ.
+
+
diff --git a/datamgmt/load/g-command-based-web-external-tables.html.md.erb b/datamgmt/load/g-command-based-web-external-tables.html.md.erb
new file mode 100644
index 0000000..7830cc3
--- /dev/null
+++ b/datamgmt/load/g-command-based-web-external-tables.html.md.erb
@@ -0,0 +1,26 @@
+---
+title: Command-based Web External Tables
+---
+
+The output of a shell command or script defines command-based web table data. Specify the command in the `EXECUTE` clause of `CREATE EXTERNAL WEB TABLE`. The data is current as of the time the command runs. The `EXECUTE` clause runs the shell command or script on the specified master or virtual segments. The virtual segments run the command in parallel. Scripts must be executable by the gpadmin user and reside in the same location on the master or the hosts of virtual segments.
+
+The command that you specify in the external table definition executes from the database and cannot access environment variables from `.bashrc` or `.profile`. Set environment variables in the `EXECUTE` clause. The following external web table, for example, runs a command on the HAWQ master host:
+
+``` sql
+CREATE EXTERNAL WEB TABLE output (output text)
+EXECUTE 'PATH=/home/gpadmin/programs; export PATH; myprogram.sh'
+ ON MASTER
+FORMAT 'TEXT';
+```
+
+The following command defines a web table that runs a script on five virtual segments.
+
+``` sql
+CREATE EXTERNAL WEB TABLE log_output (linenum int, message text)
+EXECUTE '/var/load_scripts/get_log_data.sh' ON 5
+FORMAT 'TEXT' (DELIMITER '|');
+```
+
+The virtual segments are selected by the resource manager at runtime.
+
+
diff --git a/datamgmt/load/g-configuration-file-format.html.md.erb b/datamgmt/load/g-configuration-file-format.html.md.erb
new file mode 100644
index 0000000..73f51a9
--- /dev/null
+++ b/datamgmt/load/g-configuration-file-format.html.md.erb
@@ -0,0 +1,66 @@
+---
+title: Configuration File Format
+---
+
+The `gpfdist` configuration file uses the YAML 1.1 document format and implements a schema for defining the transformation parameters. The configuration file must be a valid YAML document.
+
+The `gpfdist` program processes the document in order and uses indentation (spaces) to determine the document hierarchy and relationships of the sections to one another. The use of white space is significant. Do not use white space for formatting and do not use tabs.
+
+The following is the basic structure of a configuration file.
+
+``` pre
+---
+VERSION: 1.0.0.1
+TRANSFORMATIONS:
+transformation_name1:
+TYPE: input | output
+COMMAND: command
+CONTENT: data | paths
+SAFE: posix-regex
+STDERR: server | console
+transformation_name2:
+TYPE: input | output
+COMMAND: command
+...
+```
+
+VERSION
+Required. The version of the `gpfdist` configuration file schema. The current version is 1.0.0.1.
+
+TRANSFORMATIONS
+Required. Begins the transformation specification section. A configuration file must have at least one transformation. When `gpfdist` receives a transformation request, it looks in this section for an entry with the matching transformation name.
+
+TYPE
+Required. Specifies the direction of transformation. Values are `input` or `output`.
+
+- `input`: `gpfdist` treats the standard output of the transformation process as a stream of records to load into HAWQ.
+- `output` <span class="ph">: </span> `gpfdist` treats the standard input of the transformation process as a stream of records from HAWQ to transform and write to the appropriate output.
+
+COMMAND
+Required. Specifies the command `gpfdist` will execute to perform the transformation.
+
+For input transformations, `gpfdist` invokes the command specified in the `CONTENT` setting. The command is expected to open the underlying file(s) as appropriate and produce one line of `TEXT` for each row to load into HAWQ />. The input transform determines whether the entire content should be converted to one row or to multiple rows.
+
+For output transformations, `gpfdist` invokes this command as specified in the `CONTENT` setting. The output command is expected to open and write to the underlying file(s) as appropriate. The output transformation determines the final placement of the converted output.
+
+CONTENT
+Optional. The values are `data` and `paths`. The default value is `data`.
+
+- When `CONTENT` specifies `data`, the text `%filename%` in the `COMMAND` section is replaced by the path to the file to read or write.
+- When `CONTENT` specifies `paths`, the text `%filename%` in the `COMMAND` section is replaced by the path to the temporary file that contains the list of files to read or write.
+
+The following is an example of a `COMMAND` section showing the text `%filename%` that is replaced.
+
+``` pre
+COMMAND: /bin/bash input_transform.sh %filename%
+```
+
+SAFE
+Optional. A `POSIX `regular expression that the paths must match to be passed to the transformation. Specify `SAFE` when there is a concern about injection or improper interpretation of paths passed to the command. The default is no restriction on paths.
+
+STDERR
+Optional.The values are `server` and `console`.
+
+This setting specifies how to handle standard error output from the transformation. The default, `server`, specifies that `gpfdist` will capture the standard error output from the transformation in a temporary file and send the first 8k of that file to HAWQ as an error message. The error message will appear as a SQL error. `Console` specifies that `gpfdist` does not redirect or transmit the standard error output from the transformation.
+
+
diff --git a/datamgmt/load/g-controlling-segment-parallelism.html.md.erb b/datamgmt/load/g-controlling-segment-parallelism.html.md.erb
new file mode 100644
index 0000000..4e0096c
--- /dev/null
+++ b/datamgmt/load/g-controlling-segment-parallelism.html.md.erb
@@ -0,0 +1,11 @@
+---
+title: Controlling Segment Parallelism
+---
+
+The `gp_external_max_segs` server configuration parameter controls the number of virtual segments that can simultaneously access a single `gpfdist` instance. The default is 64. You can set the number of segments such that some segments process external data files and some perform other database processing. Set this parameter in the `hawq-site.xml` file of your master instance.
+
+The number of segments in the `gpfdist` location list specify the minimum number of virtual segments required to serve data to a `gpfdist` external table.
+
+The `hawq_rm_nvseg_perquery_perseg_limit` and `hawq_rm_nvseg_perquery_limit` parameters also control segment parallelism by specifying the maximum number of segments used in running queries on a `gpfdist` external table on the cluster.
+
+
diff --git a/datamgmt/load/g-create-an-error-table-and-declare-a-reject-limit.html.md.erb b/datamgmt/load/g-create-an-error-table-and-declare-a-reject-limit.html.md.erb
new file mode 100644
index 0000000..ade14ea
--- /dev/null
+++ b/datamgmt/load/g-create-an-error-table-and-declare-a-reject-limit.html.md.erb
@@ -0,0 +1,11 @@
+---
+title: Capture Row Formatting Errors and Declare a Reject Limit
+---
+
+The following SQL fragment captures formatting errors internally in HAWQ and declares a reject limit of 10 rows.
+
+``` sql
+LOG ERRORS INTO errortable SEGMENT REJECT LIMIT 10 ROWS
+```
+
+
diff --git a/datamgmt/load/g-creating-and-using-web-external-tables.html.md.erb b/datamgmt/load/g-creating-and-using-web-external-tables.html.md.erb
new file mode 100644
index 0000000..4ef6cab
--- /dev/null
+++ b/datamgmt/load/g-creating-and-using-web-external-tables.html.md.erb
@@ -0,0 +1,13 @@
+---
+title: Creating and Using Web External Tables
+---
+
+`CREATE EXTERNAL WEB TABLE` creates a web table definition. Web external tables allow HAWQ to treat dynamic data sources like regular database tables. Because web table data can change as a query runs, the data is not rescannable.
+
+You can define command-based or URL-based web external tables. The definition forms are distinct: you cannot mix command-based and URL-based definitions.
+
+- **[Command-based Web External Tables](../../datamgmt/load/g-command-based-web-external-tables.html)**
+
+- **[URL-based Web External Tables](../../datamgmt/load/g-url-based-web-external-tables.html)**
+
+
diff --git a/datamgmt/load/g-define-an-external-table-with-single-row-error-isolation.html.md.erb b/datamgmt/load/g-define-an-external-table-with-single-row-error-isolation.html.md.erb
new file mode 100644
index 0000000..e0c3c17
--- /dev/null
+++ b/datamgmt/load/g-define-an-external-table-with-single-row-error-isolation.html.md.erb
@@ -0,0 +1,24 @@
+---
+title: Define an External Table with Single Row Error Isolation
+---
+
+The following example logs errors internally in HAWQ and sets an error threshold of 10 errors.
+
+``` sql
+=# CREATE EXTERNAL TABLE ext_expenses ( name text, date date, amount float4, category text, desc1 text )
+ LOCATION ('gpfdist://etlhost-1:8081/*', 'gpfdist://etlhost-2:8082/*')
+ FORMAT 'TEXT' (DELIMITER '|')
+ LOG ERRORS INTO errortable SEGMENT REJECT LIMIT 10 ROWS;
+```
+
+The following example creates an external table, *ext\_expenses*, sets an error threshold of 10 errors, and writes error rows to the table *err\_expenses*.
+
+``` sql
+=# CREATE EXTERNAL TABLE ext_expenses
+ ( name text, date date, amount float4, category text, desc1 text )
+ LOCATION ('gpfdist://etlhost-1:8081/*', 'gpfdist://etlhost-2:8082/*')
+ FORMAT 'TEXT' (DELIMITER '|')
+ LOG ERRORS INTO err_expenses SEGMENT REJECT LIMIT 10 ROWS;
+```
+
+
diff --git a/datamgmt/load/g-defining-a-command-based-writable-external-web-table.html.md.erb b/datamgmt/load/g-defining-a-command-based-writable-external-web-table.html.md.erb
new file mode 100644
index 0000000..8a24474
--- /dev/null
+++ b/datamgmt/load/g-defining-a-command-based-writable-external-web-table.html.md.erb
@@ -0,0 +1,43 @@
+---
+title: Defining a Command-Based Writable External Web Table
+---
+
+You can define writable external web tables to send output rows to an application or script. The application must accept an input stream, reside in the same location on all of the HAWQ segment hosts, and be executable by the `gpadmin` user. All segments in the HAWQ system run the application or script, whether or not a segment has output rows to process.
+
+Use `CREATE WRITABLE EXTERNAL WEB TABLE` to define the external table and specify the application or script to run on the segment hosts. Commands execute from within the database and cannot access environment variables (such as `$PATH`). Set environment variables in the `EXECUTE` clause of your writable external table definition. For example:
+
+``` sql
+=# CREATE WRITABLE EXTERNAL WEB TABLE output (output text)
+ EXECUTE 'export PATH=$PATH:/home/gpadmin/programs; myprogram.sh'
+ ON 6
+ FORMAT 'TEXT'
+ DISTRIBUTED RANDOMLY;
+```
+
+The following HAWQ variables are available for use in OS commands executed by a web or writable external table. Set these variables as environment variables in the shell that executes the command(s). They can be used to identify a set of requests made by an external table statement across the HAWQ array of hosts and segment instances.
+
+<caption><span class="tablecap">Table 1. External Table EXECUTE Variables</span></caption>
+
+<a id="topic71__du224024"></a>
+
+| Variable | Description |
+|---------------------|----------------------------------------------------------------------------------------------------------------------------|
+| $GP\_CID | Command count of the transaction executing the external table statement. |
+| $GP\_DATABASE | The database in which the external table definition resides. |
+| $GP\_DATE | The date on which the external table command ran. |
+| $GP\_MASTER\_HOST | The host name of the HAWQ master host from which the external table statement was dispatched. |
+| $GP\_MASTER\_PORT | The port number of the HAWQ master instance from which the external table statement was dispatched. |
+| $GP\_SEG\_DATADIR | The location of the data directory of the segment instance executing the external table command. |
+| $GP\_SEG\_PG\_CONF | The location of the `hawq-site.xml` file of the segment instance executing the external table command. |
+| $GP\_SEG\_PORT | The port number of the segment instance executing the external table command. |
+| $GP\_SEGMENT\_COUNT | The total number of segment instances in the HAWQ system. |
+| $GP\_SEGMENT\_ID | The ID number of the segment instance executing the external table command (same as `dbid` in `gp_segment_configuration`). |
+| $GP\_SESSION\_ID | The database session identifier number associated with the external table statement. |
+| $GP\_SN | Serial number of the external table scan node in the query plan of the external table statement. |
+| $GP\_TIME | The time the external table command was executed. |
+| $GP\_USER | The database user executing the external table statement. |
+| $GP\_XID | The transaction ID of the external table statement. |
+
+- **[Disabling EXECUTE for Web or Writable External Tables](../../datamgmt/load/g-disabling-execute-for-web-or-writable-external-tables.html)**
+
+
diff --git a/datamgmt/load/g-defining-a-file-based-writable-external-table.html.md.erb b/datamgmt/load/g-defining-a-file-based-writable-external-table.html.md.erb
new file mode 100644
index 0000000..a655b07
--- /dev/null
+++ b/datamgmt/load/g-defining-a-file-based-writable-external-table.html.md.erb
@@ -0,0 +1,16 @@
+---
+title: Defining a File-Based Writable External Table
+---
+
+Writable external tables that output data to files use the HAWQ parallel file server program, `gpfdist`, or HAWQ Extensions Framework (PXF).
+
+Use the `CREATE WRITABLE EXTERNAL TABLE` command to define the external table and specify the location and format of the output files.
+
+- With a writable external table using the `gpfdist` protocol, the HAWQ segments send their data to `gpfdist`, which writes the data to the named file. `gpfdist` must run on a host that the HAWQ segments can access over the network. `gpfdist` points to a file location on the output host and writes data received from the HAWQ segments to the file. To divide the output data among multiple files, list multiple `gpfdist` URIs in your writable external table definition.
+- A writable external web table sends data to an application as a stream of data. For example, unload data from HAWQ and send it to an application that connects to another database or ETL tool to load the data elsewhere. Writable external web tables use the `EXECUTE` clause to specify a shell command, script, or application to run on the segment hosts and accept an input stream of data. See [Defining a Command-Based Writable External Web Table](g-defining-a-command-based-writable-external-web-table.html#topic71) for more information about using `EXECUTE` commands in a writable external table definition.
+
+You can optionally declare a distribution policy for your writable external tables. By default, writable external tables use a random distribution policy. If the source table you are exporting data from has a hash distribution policy, defining the same distribution key column(s) for the writable external table improves unload performance by eliminating the requirement to move rows over the interconnect. If you unload data from a particular table, you can use the `LIKE` clause to copy the column definitions and distribution policy from the source table.
+
+- **[Example - HAWQ file server (gpfdist)](../../datamgmt/load/g-example-greenplum-file-server-gpfdist.html)**
+
+
diff --git a/datamgmt/load/g-determine-the-transformation-schema.html.md.erb b/datamgmt/load/g-determine-the-transformation-schema.html.md.erb
new file mode 100644
index 0000000..1a4eb9b
--- /dev/null
+++ b/datamgmt/load/g-determine-the-transformation-schema.html.md.erb
@@ -0,0 +1,33 @@
+---
+title: Determine the Transformation Schema
+---
+
+To prepare for the transformation project:
+
+1. <span class="ph">Determine the goal of the project, such as indexing data, analyzing data, combining data, and so on.</span>
+2. <span class="ph">Examine the XML file and note the file structure and element names. </span>
+3. <span class="ph">Choose the elements to import and decide if any other limits are appropriate. </span>
+
+For example, the following XML file, *prices.xml*, is a simple, short file that contains price records. Each price record contains two fields: an item number and a price.
+
+``` xml
+<?xml version="1.0" encoding="ISO-8859-1" ?>
+<prices>
+ <pricerecord>
+ <itemnumber>708421</itemnumber>
+ <price>19.99</price>
+ </pricerecord>
+ <pricerecord>
+ <itemnumber>708466</itemnumber>
+ <price>59.25</price>
+ </pricerecord>
+ <pricerecord>
+ <itemnumber>711121</itemnumber>
+ <price>24.99</price>
+ </pricerecord>
+</prices>
+```
+
+The goal is to import all the data into a HAWQ table with an integer `itemnumber` column and a decimal `price` column.
+
+
diff --git a/datamgmt/load/g-disabling-execute-for-web-or-writable-external-tables.html.md.erb b/datamgmt/load/g-disabling-execute-for-web-or-writable-external-tables.html.md.erb
new file mode 100644
index 0000000..f0332b5
--- /dev/null
+++ b/datamgmt/load/g-disabling-execute-for-web-or-writable-external-tables.html.md.erb
@@ -0,0 +1,11 @@
+---
+title: Disabling EXECUTE for Web or Writable External Tables
+---
+
+There is a security risk associated with allowing external tables to execute OS commands or scripts. To disable the use of `EXECUTE` in web and writable external table definitions, set the `gp_external_enable_exec server` configuration parameter to off in your master `hawq-site.xml` file:
+
+``` pre
+gp_external_enable_exec = off
+```
+
+
diff --git a/datamgmt/load/g-escaping-in-csv-formatted-files.html.md.erb b/datamgmt/load/g-escaping-in-csv-formatted-files.html.md.erb
new file mode 100644
index 0000000..d07b463
--- /dev/null
+++ b/datamgmt/load/g-escaping-in-csv-formatted-files.html.md.erb
@@ -0,0 +1,29 @@
+---
+title: Escaping in CSV Formatted Files
+---
+
+By default, the escape character is a `"` (double quote) for CSV-formatted files. If you want to use a different escape character, use the `ESCAPE` clause of `COPY`, `CREATE EXTERNAL TABLE` or the `hawq load` control file to declare a different escape character. In cases where your selected escape character is present in your data, you can use it to escape itself.
+
+For example, suppose you have a table with three columns and you want to load the following three fields:
+
+- `Free trip to A,B`
+- `5.89`
+- `Special rate "1.79"`
+
+Your designated delimiter character is `,` (comma), and your designated escape character is `"` (double quote). The formatted row in your data file looks like this:
+
+``` pre
+ "Free trip to A,B","5.89","Special rate ""1.79"""
+
+
+```
+
+The data value with a comma character that is part of the data is enclosed in double quotes. The double quotes that are part of the data are escaped with a double quote even though the field value is enclosed in double quotes.
+
+Embedding the entire field inside a set of double quotes guarantees preservation of leading and trailing whitespace characters:
+
+`"`Free trip to A,B `"`,`"`5.89 `"`,`"`Special rate `""`1.79`"" "`
+
+**Note:** In CSV mode, all characters are significant. A quoted value surrounded by white space, or any characters other than `DELIMITER`, includes those characters. This can cause errors if you import data from a system that pads CSV lines with white space to some fixed width. In this case, preprocess the CSV file to remove the trailing white space before importing the data into HAWQ.
+
+
diff --git a/datamgmt/load/g-escaping-in-text-formatted-files.html.md.erb b/datamgmt/load/g-escaping-in-text-formatted-files.html.md.erb
new file mode 100644
index 0000000..e24a2b7
--- /dev/null
+++ b/datamgmt/load/g-escaping-in-text-formatted-files.html.md.erb
@@ -0,0 +1,31 @@
+---
+title: Escaping in Text Formatted Files
+---
+
+By default, the escape character is a \\ (backslash) for text-formatted files. You can declare a different escape character in the `ESCAPE` clause of `COPY`, `CREATE EXTERNAL TABLE`, or the `hawq load` control file. If your escape character appears in your data, use it to escape itself.
+
+For example, suppose you have a table with three columns and you want to load the following three fields:
+
+- `backslash = \`
+- `vertical bar = |`
+- `exclamation point = !`
+
+Your designated delimiter character is `|` (pipe character), and your designated escape character is `\` (backslash). The formatted row in your data file looks like this:
+
+``` pre
+backslash = \\ | vertical bar = \| | exclamation point = !
+```
+
+Notice how the backslash character that is part of the data is escaped with another backslash character, and the pipe character that is part of the data is escaped with a backslash character.
+
+You can use the escape character to escape octal and hexidecimal sequences. The escaped value is converted to the equivalent character when loaded into HAWQ. For example, to load the ampersand character (`&`), use the escape character to escape its equivalent hexidecimal (`\0x26`) or octal (`\046`) representation.
+
+You can disable escaping in `TEXT`-formatted files using the `ESCAPE` clause of `COPY`, `CREATE EXTERNAL TABLE` or the `hawq load` control file as follows:
+
+``` pre
+ESCAPE 'OFF'
+```
+
+This is useful for input data that contains many backslash characters, such as web log data.
+
+
diff --git a/datamgmt/load/g-escaping.html.md.erb b/datamgmt/load/g-escaping.html.md.erb
new file mode 100644
index 0000000..0a1e62a
--- /dev/null
+++ b/datamgmt/load/g-escaping.html.md.erb
@@ -0,0 +1,16 @@
+---
+title: Escaping
+---
+
+There are two reserved characters that have special meaning to HAWQ:
+
+- The designated delimiter character separates columns or fields in the data file.
+- The newline character designates a new row in the data file.
+
+If your data contains either of these characters, you must escape the character so that HAWQ treats it as data and not as a field separator or new row. By default, the escape character is a \\ (backslash) for text-formatted files and a double quote (") for csv-formatted files.
+
+- **[Escaping in Text Formatted Files](../../datamgmt/load/g-escaping-in-text-formatted-files.html)**
+
+- **[Escaping in CSV Formatted Files](../../datamgmt/load/g-escaping-in-csv-formatted-files.html)**
+
+
diff --git a/datamgmt/load/g-example-1-dblp-database-publications-in-demo-directory.html.md.erb b/datamgmt/load/g-example-1-dblp-database-publications-in-demo-directory.html.md.erb
new file mode 100644
index 0000000..4f61396
--- /dev/null
+++ b/datamgmt/load/g-example-1-dblp-database-publications-in-demo-directory.html.md.erb
@@ -0,0 +1,29 @@
+---
+title: Command-based Web External Tables
+---
+
+The output of a shell command or script defines command-based web table data. Specify the command in the `EXECUTE` clause of `CREATE EXTERNAL WEB TABLE`. The data is current as of the time the command runs. The `EXECUTE` clause runs the shell command or script on the specified master, and/or segment host or hosts. The command or script must reside on the hosts corresponding to the host(s) defined in the `EXECUTE` clause.
+
+By default, the command is run on segment hosts when active segments have output rows to process. For example, if each segment host runs four primary segment instances that have output rows to process, the command runs four times per segment host. You can optionally limit the number of segment instances that execute the web table command. All segments included in the web table definition in the `ON` clause run the command in parallel.
+
+The command that you specify in the external table definition executes from the database and cannot access environment variables from `.bashrc` or `.profile`. Set environment variables in the `EXECUTE` clause. For example:
+
+``` sql
+=# CREATE EXTERNAL WEB TABLE output (output text)
+EXECUTE 'PATH=/home/gpadmin/programs; export PATH; myprogram.sh'
+ ON MASTER
+FORMAT 'TEXT';
+```
+
+Scripts must be executable by the `gpadmin` user and reside in the same location on the master or segment hosts.
+
+The following command defines a web table that runs a script. The script runs on five virtual segments selected by the resource manager at runtime.
+
+``` sql
+=# CREATE EXTERNAL WEB TABLE log_output
+(linenum int, message text)
+EXECUTE '/var/load_scripts/get_log_data.sh' ON 5
+FORMAT 'TEXT' (DELIMITER '|');
+```
+
+
diff --git a/datamgmt/load/g-example-greenplum-file-server-gpfdist.html.md.erb b/datamgmt/load/g-example-greenplum-file-server-gpfdist.html.md.erb
new file mode 100644
index 0000000..a0bf669
--- /dev/null
+++ b/datamgmt/load/g-example-greenplum-file-server-gpfdist.html.md.erb
@@ -0,0 +1,13 @@
+---
+title: Example - HAWQ file server (gpfdist)
+---
+
+``` sql
+=# CREATE WRITABLE EXTERNAL TABLE unload_expenses
+( LIKE expenses )
+LOCATION ('gpfdist://etlhost-1:8081/expenses1.out',
+'gpfdist://etlhost-2:8081/expenses2.out')
+FORMAT 'TEXT' (DELIMITER ',');
+```
+
+
diff --git a/datamgmt/load/g-example-irs-mef-xml-files-in-demo-directory.html.md.erb b/datamgmt/load/g-example-irs-mef-xml-files-in-demo-directory.html.md.erb
new file mode 100644
index 0000000..6f5b9e3
--- /dev/null
+++ b/datamgmt/load/g-example-irs-mef-xml-files-in-demo-directory.html.md.erb
@@ -0,0 +1,54 @@
+---
+title: Example using IRS MeF XML Files (In demo Directory)
+---
+
+This example demonstrates loading a sample IRS Modernized eFile tax return using a Joost STX transformation. The data is in the form of a complex XML file.
+
+The U.S. Internal Revenue Service (IRS) made a significant commitment to XML and specifies its use in its Modernized e-File (MeF) system. In MeF, each tax return is an XML document with a deep hierarchical structure that closely reflects the particular form of the underlying tax code.
+
+XML, XML Schema and stylesheets play a role in their data representation and business workflow. The actual XML data is extracted from a ZIP file attached to a MIME "transmission file" message. For more information about MeF, see [Modernized e-File (Overview)](http://www.irs.gov/uac/Modernized-e-File-Overview) on the IRS web site.
+
+The sample XML document, *RET990EZ\_2006.xml*, is about 350KB in size with two elements:
+
+- ReturnHeader
+- ReturnData
+
+The <ReturnHeader> element contains general details about the tax return such as the taxpayer's name, the tax year of the return, and the preparer. The <ReturnData> element contains multiple sections with specific details about the tax return and associated schedules.
+
+The following is an abridged sample of the XML file.
+
+``` xml
+<?xml version="1.0" encoding="UTF-8"?>
+<Return returnVersion="2006v2.0"
+ xmlns="http://www.irs.gov/efile"
+ xmlns:efile="http://www.irs.gov/efile"
+ xsi:schemaLocation="http://www.irs.gov/efile"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
+ <ReturnHeader binaryAttachmentCount="1">
+ <ReturnId>AAAAAAAAAAAAAAAAAAAA</ReturnId>
+ <Timestamp>1999-05-30T12:01:01+05:01</Timestamp>
+ <ReturnType>990EZ</ReturnType>
+ <TaxPeriodBeginDate>2005-01-01</TaxPeriodBeginDate>
+ <TaxPeriodEndDate>2005-12-31</TaxPeriodEndDate>
+ <Filer>
+ <EIN>011248772</EIN>
+ ... more data ...
+ </Filer>
+ <Preparer>
+ <Name>Percy Polar</Name>
+ ... more data ...
+ </Preparer>
+ <TaxYear>2005</TaxYear>
+ </ReturnHeader>
+ ... more data ..
+```
+
+The goal is to import all the data into a HAWQ database. First, convert the XML document into text with newlines "escaped", with two columns: `ReturnId` and a single column on the end for the entire MeF tax return. For example:
+
+``` pre
+AAAAAAAAAAAAAAAAAAAA|<Return returnVersion="2006v2.0"...
+```
+
+Load the data into HAWQ.
+
+
diff --git a/datamgmt/load/g-example-witsml-files-in-demo-directory.html.md.erb b/datamgmt/load/g-example-witsml-files-in-demo-directory.html.md.erb
new file mode 100644
index 0000000..22095ef
--- /dev/null
+++ b/datamgmt/load/g-example-witsml-files-in-demo-directory.html.md.erb
@@ -0,0 +1,54 @@
+---
+title: Example using WITSML™ Files (In demo Directory)
+---
+
+This example demonstrates loading sample data describing an oil rig using a Joost STX transformation. The data is in the form of a complex XML file downloaded from energistics.org.
+
+The Wellsite Information Transfer Standard Markup Language (WITSML™) is an oil industry initiative to provide open, non-proprietary, standard interfaces for technology and software to share information among oil companies, service companies, drilling contractors, application vendors, and regulatory agencies. For more information about WITSML™, see [http://www.witsml.org](http://www.witsml.org).
+
+The oil rig information consists of a top level `<rigs>` element with multiple child elements such as `<documentInfo>, <rig>`, and so on. The following excerpt from the file shows the type of information in the `<rig>` tag.
+
+``` xml
+<?xml version="1.0" encoding="UTF-8"?>
+<?xml-stylesheet href="../stylesheets/rig.xsl" type="text/xsl" media="screen"?>
+<rigs
+ xmlns="http://www.witsml.org/schemas/131"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://www.witsml.org/schemas/131 ../obj_rig.xsd"
+ version="1.3.1.1">
+ <documentInfo>
+ ... misc data ...
+ </documentInfo>
+ <rig uidWell="W-12" uidWellbore="B-01" uid="xr31">
+ <nameWell>6507/7-A-42</nameWell>
+ <nameWellbore>A-42</nameWellbore>
+ <name>Deep Drill #5</name>
+ <owner>Deep Drilling Co.</owner>
+ <typeRig>floater</typeRig>
+ <manufacturer>Fitsui Engineering</manufacturer>
+ <yearEntService>1980</yearEntService>
+ <classRig>ABS Class A1 M CSDU AMS ACCU</classRig>
+ <approvals>DNV</approvals>
+ ... more data ...
+```
+
+The goal is to import the information for this rig into HAWQ.
+
+The sample document, *rig.xml*, is about 11KB in size. The input does not contain tabs so the relevant information can be converted into records delimited with a pipe (|).
+
+`W-12|6507/7-A-42|xr31|Deep Drill #5|Deep Drilling Co.|John Doe|John.Doe@his-ISP.com|`
+
+With the columns:
+
+- `well_uid text`, -- e.g. W-12
+- `well_name text`, -- e.g. 6507/7-A-42
+- `rig_uid text`, -- e.g. xr31
+- `rig_name text`, -- e.g. Deep Drill \#5
+- `rig_owner text`, -- e.g. Deep Drilling Co.
+- `rig_contact text`, -- e.g. John Doe
+- `rig_email text`, -- e.g. John.Doe@his-ISP.com
+- `doc xml`
+
+Then, load the data into HAWQ.
+
+
diff --git a/datamgmt/load/g-examples-read-fixed-width-data.html.md.erb b/datamgmt/load/g-examples-read-fixed-width-data.html.md.erb
new file mode 100644
index 0000000..174529a
--- /dev/null
+++ b/datamgmt/load/g-examples-read-fixed-width-data.html.md.erb
@@ -0,0 +1,37 @@
+---
+title: Examples - Read Fixed-Width Data
+---
+
+The following examples show how to read fixed-width data.
+
+## Example 1 – Loading a table with PRESERVED\_BLANKS on
+
+``` sql
+CREATE READABLE EXTERNAL TABLE students (
+ name varchar(20), address varchar(30), age int)
+LOCATION ('gpfdist://host:port/file/path/')
+FORMAT 'CUSTOM' (formatter=fixedwidth_in, name=20, address=30, age=4,
+ preserve_blanks='on',null='NULL');
+```
+
+## Example 2 – Loading data with no line delimiter
+
+``` sql
+CREATE READABLE EXTERNAL TABLE students (
+ name varchar(20), address varchar(30), age int)
+LOCATION ('gpfdist://host:port/file/path/')
+FORMAT 'CUSTOM' (formatter=fixedwidth_in, name='20', address='30', age='4',
+ line_delim='?@');
+```
+
+## Example 3 – Create a writable external table with a \\r\\n line delimiter
+
+``` sql
+CREATE WRITABLE EXTERNAL TABLE students_out (
+ name varchar(20), address varchar(30), age int)
+LOCATION ('gpfdist://host:port/file/path/filename')
+FORMAT 'CUSTOM' (formatter=fixedwidth_out,
+ name=20, address=30, age=4, line_delim=E'\r\n');
+```
+
+
diff --git a/datamgmt/load/g-external-tables.html.md.erb b/datamgmt/load/g-external-tables.html.md.erb
new file mode 100644
index 0000000..4142a07
--- /dev/null
+++ b/datamgmt/load/g-external-tables.html.md.erb
@@ -0,0 +1,44 @@
+---
+title: Accessing File-Based External Tables
+---
+
+External tables enable accessing external files as if they are regular database tables. They are often used to move data into and out of a HAWQ database.
+
+To create an external table definition, you specify the format of your input files and the location of your external data sources. For information input file formats, see [Formatting Data Files](g-formatting-data-files.html#topic95).
+
+Use one of the following protocols to access external table data sources. You cannot mix protocols in `CREATE EXTERNAL TABLE` statements:
+
+- `gpfdist://` points to a directory on the file host and serves external data files to all HAWQ segments in parallel. See [gpfdist Protocol](g-gpfdist-protocol.html#topic_sny_yph_kr).
+- `gpfdists://` is the secure version of `gpfdist`. See [gpfdists Protocol](g-gpfdists-protocol.html#topic_sny_yph_kr).
+- `pxf://` specifies data accessed through the HAWQ Extensions Framework (PXF). PXF is a service that uses plug-in Java classes to read and write data in external data sources. PXF includes plug-ins to access data in HDFS, HBase, and Hive. Custom plug-ins can be written to access other external data sources.
+
+External tables allow you to access external files from within the database as if they are regular database tables. Used with `gpfdist`, the HAWQ parallel file distribution program, or HAWQ Extensions Framework (PXF), external tables provide full parallelism by using the resources of all HAWQ segments to load or unload data.
+
+You can query external table data directly and in parallel using SQL commands such as `SELECT`, `JOIN`, or `SORT EXTERNAL TABLE DATA`, and you can create views for external tables.
+
+The steps for using external tables are:
+
+1. Define the external table.
+2. Start the gpfdist file server(s) if you plan to use the `gpfdist` or `gpdists` protocols.
+3. Place the data files in the correct locations.
+4. Query the external table with SQL commands.
+
+HAWQ provides readable and writable external tables:
+
+- Readable external tables for data loading. Readable external tables support basic extraction, transformation, and loading (ETL) tasks common in data warehousing. HAWQ segment instances read external table data in parallel to optimize large load operations. You cannot modify readable external tables.
+- Writable external tables for data unloading. Writable external tables support:
+
+ - Selecting data from database tables to insert into the writable external table.
+ - Sending data to an application as a stream of data. For example, unload data from HAWQ and send it to an application that connects to another database or ETL tool to load the data elsewhere.
+ - Receiving output from HAWQ parallel MapReduce calculations.
+
+ Writable external tables allow only `INSERT` operations.
+
+External tables can be file-based or web-based.
+
+- Regular (file-based) external tables access static flat files. Regular external tables are rescannable: the data is static while the query runs.
+- Web (web-based) external tables access dynamic data sources, either on a web server with the `http://` protocol or by executing OS commands or scripts. Web external tables are not rescannable: the data can change while the query runs.
+
+Dump and restore operate only on external and web external table *definitions*, not on the data sources.
+
+
diff --git a/datamgmt/load/g-formatting-columns.html.md.erb b/datamgmt/load/g-formatting-columns.html.md.erb
new file mode 100644
index 0000000..b828212
--- /dev/null
+++ b/datamgmt/load/g-formatting-columns.html.md.erb
@@ -0,0 +1,19 @@
+---
+title: Formatting Columns
+---
+
+The default column or field delimiter is the horizontal `TAB` character (`0x09`) for text files and the comma character (`0x2C`) for CSV files. You can declare a single character delimiter using the `DELIMITER` clause of `COPY`, `CREATE EXTERNAL TABLE` or the `hawq load` configuration table when you define your data format. The delimiter character must appear between any two data value fields. Do not place a delimiter at the beginning or end of a row. For example, if the pipe character ( | ) is your delimiter:
+
+``` pre
+data value 1|data value 2|data value 3
+```
+
+The following command shows the use of the pipe character as a column delimiter:
+
+``` sql
+=# CREATE EXTERNAL TABLE ext_table (name text, date date)
+LOCATION ('gpfdist://host:port/filename.txt)
+FORMAT 'TEXT' (DELIMITER '|');
+```
+
+
diff --git a/datamgmt/load/g-formatting-data-files.html.md.erb b/datamgmt/load/g-formatting-data-files.html.md.erb
new file mode 100644
index 0000000..6c929ad
--- /dev/null
+++ b/datamgmt/load/g-formatting-data-files.html.md.erb
@@ -0,0 +1,17 @@
+---
+title: Formatting Data Files
+---
+
+When you use the HAWQ tools for loading and unloading data, you must specify how your data is formatted. `COPY`, `CREATE EXTERNAL TABLE, `and `hawq load` have clauses that allow you to specify how your data is formatted. Data can be delimited text (`TEXT`) or comma separated values (`CSV`) format. External data must be formatted correctly to be read by HAWQ. This topic explains the format of data files expected by HAWQ.
+
+- **[Formatting Rows](../../datamgmt/load/g-formatting-rows.html)**
+
+- **[Formatting Columns](../../datamgmt/load/g-formatting-columns.html)**
+
+- **[Representing NULL Values](../../datamgmt/load/g-representing-null-values.html)**
+
+- **[Escaping](../../datamgmt/load/g-escaping.html)**
+
+- **[Character Encoding](../../datamgmt/load/g-character-encoding.html)**
+
+
diff --git a/datamgmt/load/g-formatting-rows.html.md.erb b/datamgmt/load/g-formatting-rows.html.md.erb
new file mode 100644
index 0000000..ea9b416
--- /dev/null
+++ b/datamgmt/load/g-formatting-rows.html.md.erb
@@ -0,0 +1,7 @@
+---
+title: Formatting Rows
+---
+
+HAWQ expects rows of data to be separated by the `LF` character (Line feed, `0x0A`), `CR` (Carriage return, `0x0D`), or `CR` followed by `LF` (`CR+LF`, `0x0D 0x0A`). `LF` is the standard newline representation on UNIX or UNIX-like operating systems. Operating systems such as Windows or Mac OS X use `CR` or `CR+LF`. All of these representations of a newline are supported by HAWQ as a row delimiter. For more information, see [Importing and Exporting Fixed Width Data](g-importing-and-exporting-fixed-width-data.html#topic37).
+
+
diff --git a/datamgmt/load/g-gpfdist-protocol.html.md.erb b/datamgmt/load/g-gpfdist-protocol.html.md.erb
new file mode 100644
index 0000000..f41c946
--- /dev/null
+++ b/datamgmt/load/g-gpfdist-protocol.html.md.erb
@@ -0,0 +1,15 @@
+---
+title: gpfdist Protocol
+---
+
+The `gpfdist://` protocol is used in a URI to reference a running `gpfdist` instance. The `gpfdist` utility serves external data files from a directory on a file host to all HAWQ segments in parallel.
+
+`gpfdist` is located in the `$GPHOME/bin` directory on your HAWQ master host and on each segment host.
+
+Run `gpfdist` on the host where the external data files reside. `gpfdist` uncompresses `gzip` (`.gz`) and `bzip2` (.`bz2`) files automatically. You can use the wildcard character (\*) or other C-style pattern matching to denote multiple files to read. The files specified are assumed to be relative to the directory that you specified when you started the `gpfdist` instance.
+
+All virtual segments access the external file(s) in parallel, subject to the number of segments set in the `gp_external_max_segments` parameter, the length of the `gpfdist` location list, and the limits specified by the `hawq_rm_nvseg_perquery_limit` and `hawq_rm_nvseg_perquery_perseg_limit` parameters. Use multiple `gpfdist` data sources in a `CREATE EXTERNAL TABLE` statement to scale the external table's scan performance. For more information about configuring `gpfdist`, see [Using the Greenplum Parallel File Server (gpfdist)](g-using-the-greenplum-parallel-file-server--gpfdist-.html#topic13).
+
+See the `gpfdist` reference documentation for more information about using `gpfdist` with external tables.
+
+
diff --git a/datamgmt/load/g-gpfdists-protocol.html.md.erb b/datamgmt/load/g-gpfdists-protocol.html.md.erb
new file mode 100644
index 0000000..2f5641d
--- /dev/null
+++ b/datamgmt/load/g-gpfdists-protocol.html.md.erb
@@ -0,0 +1,37 @@
+---
+title: gpfdists Protocol
+---
+
+The `gpfdists://` protocol is a secure version of the `gpfdist:// protocol`. To use it, you run the `gpfdist` utility with the `--ssl` option. When specified in a URI, the `gpfdists://` protocol enables encrypted communication and secure identification of the file server and the HAWQ to protect against attacks such as eavesdropping and man-in-the-middle attacks.
+
+`gpfdists` implements SSL security in a client/server scheme with the following attributes and limitations:
+
+- Client certificates are required.
+- Multilingual certificates are not supported.
+- A Certificate Revocation List (CRL) is not supported.
+- The `TLSv1` protocol is used with the `TLS_RSA_WITH_AES_128_CBC_SHA` encryption algorithm.
+- SSL parameters cannot be changed.
+- SSL renegotiation is supported.
+- The SSL ignore host mismatch parameter is set to `false`.
+- Private keys containing a passphrase are not supported for the `gpfdist` file server (server.key) and for the HAWQ (client.key).
+- Issuing certificates that are appropriate for the operating system in use is the user's responsibility. Generally, converting certificates as shown in [https://www.sslshopper.com/ssl-converter.html](https://www.sslshopper.com/ssl-converter.html) is supported.
+
+ **Note:** A server started with the `gpfdist --ssl` option can only communicate with the `gpfdists` protocol. A server that was started with `gpfdist` without the `--ssl` option can only communicate with the `gpfdist` protocol.
+
+- The client certificate file, client.crt
+- The client private key file, client.key
+
+Use one of the following methods to invoke the `gpfdists` protocol.
+
+- Run `gpfdist` with the `--ssl` option and then use the `gpfdists` protocol in the `LOCATION` clause of a `CREATE EXTERNAL TABLE` statement.
+- Use a `hawq load` YAML control file with the `SSL` option set to true.
+
+Using `gpfdists` requires that the following client certificates reside in the `$PGDATA/gpfdists` directory on each segment.
+
+- The client certificate file, `client.crt`
+- The client private key file, `client.key`
+- The trusted certificate authorities, `root.crt`
+
+For an example of loading data into an external table security, see [Example 3 - Multiple gpfdists instances](creating-external-tables-examples.html#topic47).
+
+
diff --git a/datamgmt/load/g-handling-errors-ext-table-data.html.md.erb b/datamgmt/load/g-handling-errors-ext-table-data.html.md.erb
new file mode 100644
index 0000000..2b8dc78
--- /dev/null
+++ b/datamgmt/load/g-handling-errors-ext-table-data.html.md.erb
@@ -0,0 +1,9 @@
+---
+title: Handling Errors in External Table Data
+---
+
+By default, if external table data contains an error, the command fails and no data loads into the target database table. Define the external table with single row error handling to enable loading correctly formatted rows and to isolate data errors in external table data. See [Handling Load Errors](g-handling-load-errors.html#topic55).
+
+The `gpfdist` file server uses the `HTTP` protocol. External table queries that use `LIMIT` end the connection after retrieving the rows, causing an HTTP socket error. If you use `LIMIT` in queries of external tables that use the `gpfdist://` or `http:// protocols`, ignore these errors – data is returned to the database as expected.
+
+
diff --git a/datamgmt/load/g-handling-load-errors.html.md.erb b/datamgmt/load/g-handling-load-errors.html.md.erb
new file mode 100644
index 0000000..6faf7a5
--- /dev/null
+++ b/datamgmt/load/g-handling-load-errors.html.md.erb
@@ -0,0 +1,28 @@
+---
+title: Handling Load Errors
+---
+
+Readable external tables are most commonly used to select data to load into regular database tables. You use the `CREATE TABLE AS SELECT` or `INSERT INTO `commands to query the external table data. By default, if the data contains an error, the entire command fails and the data is not loaded into the target database table.
+
+The `SEGMENT REJECT LIMIT` clause allows you to isolate format errors in external table data and to continue loading correctly formatted rows. Use `SEGMENT REJECT LIMIT `to set an error threshold, specifying the reject limit `count` as number of `ROWS` (the default) or as a `PERCENT` of total rows (1-100).
+
+The entire external table operation is aborted, and no rows are processed, if the number of error rows reaches the `SEGMENT REJECT LIMIT`. The limit of error rows is per-segment, not per entire operation. The operation processes all good rows, and it discards and optionally logs formatting errors for erroneous rows, if the number of error rows does not reach the `SEGMENT REJECT LIMIT`.
+
+The `LOG ERRORS` clause allows you to keep error rows for further examination. For information about the `LOG ERRORS` clause, see the `CREATE EXTERNAL TABLE` command.
+
+When you set `SEGMENT REJECT LIMIT`, HAWQ scans the external data in single row error isolation mode. Single row error isolation mode applies to external data rows with format errors such as extra or missing attributes, attributes of a wrong data type, or invalid client encoding sequences. HAWQ does not check constraint errors, but you can filter constraint errors by limiting the `SELECT` from an external table at runtime. For example, to eliminate duplicate key errors:
+
+``` sql
+=# INSERT INTO table_with_pkeys
+SELECT DISTINCT * FROM external_table;
+```
+
+- **[Define an External Table with Single Row Error Isolation](../../datamgmt/load/g-define-an-external-table-with-single-row-error-isolation.html)**
+
+- **[Capture Row Formatting Errors and Declare a Reject Limit](../../datamgmt/load/g-create-an-error-table-and-declare-a-reject-limit.html)**
+
+- **[Identifying Invalid CSV Files in Error Table Data](../../datamgmt/load/g-identifying-invalid-csv-files-in-error-table-data.html)**
+
+- **[Moving Data between Tables](../../datamgmt/load/g-moving-data-between-tables.html)**
+
+
diff --git a/datamgmt/load/g-identifying-invalid-csv-files-in-error-table-data.html.md.erb b/datamgmt/load/g-identifying-invalid-csv-files-in-error-table-data.html.md.erb
new file mode 100644
index 0000000..534d530
--- /dev/null
+++ b/datamgmt/load/g-identifying-invalid-csv-files-in-error-table-data.html.md.erb
@@ -0,0 +1,7 @@
+---
+title: Identifying Invalid CSV Files in Error Table Data
+---
+
+If a CSV file contains invalid formatting, the *rawdata* field in the error table can contain several combined rows. For example, if a closing quote for a specific field is missing, all the following newlines are treated as embedded newlines. When this happens, HAWQ stops parsing a row when it reaches 64K, puts that 64K of data into the error table as a single row, resets the quote flag, and continues. If this happens three times during load processing, the load file is considered invalid and the entire load fails with the message "`rejected ` `N` ` or more rows`". See [Escaping in CSV Formatted Files](g-escaping-in-csv-formatted-files.html#topic101) for more information on the correct use of quotes in CSV files.
+
+
diff --git a/datamgmt/load/g-importing-and-exporting-fixed-width-data.html.md.erb b/datamgmt/load/g-importing-and-exporting-fixed-width-data.html.md.erb
new file mode 100644
index 0000000..f49cae0
--- /dev/null
+++ b/datamgmt/load/g-importing-and-exporting-fixed-width-data.html.md.erb
@@ -0,0 +1,38 @@
+---
+title: Importing and Exporting Fixed Width Data
+---
+
+Specify custom formats for fixed-width data with the HAWQ functions `fixedwith_in` and `fixedwidth_out`. These functions already exist in the file `$GPHOME/share/postgresql/cdb_external_extensions.sql`. The following example declares a custom format, then calls the `fixedwidth_in` function to format the data.
+
+``` sql
+CREATE READABLE EXTERNAL TABLE students (
+ name varchar(20), address varchar(30), age int)
+LOCATION ('gpfdist://mdw:8081/students.txt')
+FORMAT 'CUSTOM' (formatter=fixedwidth_in, name='20', address='30', age='4');
+```
+
+The following options specify how to import fixed width data.
+
+- Read all the data.
+
+ To load all the fields on a line of fixed with data, you must load them in their physical order. You must specify the field length, but cannot specify a starting and ending position. The fields names in the fixed width arguments must match the order in the field list at the beginning of the `CREATE TABLE` command.
+
+- Set options for blank and null characters.
+
+ Trailing blanks are trimmed by default. To keep trailing blanks, use the `preserve_blanks=on` option.You can reset the trailing blanks option to the default with the `preserve_blanks=off` option.
+
+ Use the null=`'null_string_value'` option to specify a value for null characters.
+
+- If you specify `preserve_blanks=on`, you must also define a value for null characters.
+- If you specify `preserve_blanks=off`, null is not defined, and the field contains only blanks, HAWQ writes a null to the table. If null is defined, HAWQ writes an empty string to the table.
+
+ Use the `line_delim='line_ending'` parameter to specify the line ending character. The following examples cover most cases. The `E` specifies an escape string constant.
+
+ ``` pre
+ line_delim=E'\n'
+ line_delim=E'\r'
+ line_delim=E'\r\n'
+ line_delim='abc'
+ ```
+
+
diff --git a/datamgmt/load/g-installing-gpfdist.html.md.erb b/datamgmt/load/g-installing-gpfdist.html.md.erb
new file mode 100644
index 0000000..25f923e
--- /dev/null
+++ b/datamgmt/load/g-installing-gpfdist.html.md.erb
@@ -0,0 +1,7 @@
+---
+title: Installing gpfdist
+---
+
+`gpfdist` is installed in `$GPHOME/bin` of your HAWQ master host installation. Run `gpfdist` from a machine other than the HAWQ master, such as on a machine devoted to ETL processing. If you want to install `gpfdist` on your ETL server, get it from the *Greenplum Load Tools* package and follow its installation instructions.
+
+
diff --git a/datamgmt/load/g-load-the-data.html.md.erb b/datamgmt/load/g-load-the-data.html.md.erb
new file mode 100644
index 0000000..4c88c9f
--- /dev/null
+++ b/datamgmt/load/g-load-the-data.html.md.erb
@@ -0,0 +1,17 @@
+---
+title: Load the Data
+---
+
+Create the tables with SQL statements based on the appropriate schema.
+
+There are no special requirements for the HAWQ tables that hold loaded data. In the prices example, the following command creates the appropriate table.
+
+``` sql
+CREATE TABLE prices (
+ itemnumber integer,
+ price decimal
+)
+DISTRIBUTED BY (itemnumber);
+```
+
+
diff --git a/datamgmt/load/g-loading-and-unloading-data.html.md.erb b/datamgmt/load/g-loading-and-unloading-data.html.md.erb
new file mode 100644
index 0000000..cbcab0b
--- /dev/null
+++ b/datamgmt/load/g-loading-and-unloading-data.html.md.erb
@@ -0,0 +1,53 @@
+---
+title: Loading and Unloading Data
+---
+
+The topics in this section describe methods for loading and writing data into and out of HAWQ, and how to format data files.
+
+HAWQ supports high-performance parallel data loading and unloading, and for smaller amounts of data, single file, non-parallel data import and export.
+
+HAWQ can read from and write to several types of external data sources, including text files, Hadoop file systems, and web servers.
+
+- The `COPY` SQL command transfers data between an external text file on the master host and a HAWQ database table.
+- External tables allow you to query data outside of the database directly and in parallel using SQL commands such as `SELECT`, `JOIN`, or `SORT EXTERNAL TABLE DATA`, and you can create views for external tables. External tables are often used to load external data into a regular database table using a command such as `CREATE TABLE table AS SELECT * FROM ext_table`.
+- External web tables provide access to dynamic data. They can be backed with data from URLs accessed using the HTTP protocol or by the output of an OS script running on one or more segments.
+- The `gpfdist` utility is the HAWQ parallel file distribution program. It is an HTTP server that is used with external tables to allow HAWQ segments to load external data in parallel, from multiple file systems. You can run multiple instances of `gpfdist` on different hosts and network interfaces and access them in parallel.
+- The `hawq load` utility automates the steps of a load task using a YAML-formatted control file.
+
+The method you choose to load data depends on the characteristics of the source data—its location, size, format, and any transformations required.
+
+In the simplest case, the `COPY` SQL command loads data into a table from a text file that is accessible to the HAWQ master instance. This requires no setup and provides good performance for smaller amounts of data. With the `COPY` command, the data copied into or out of the database passes between a single file on the master host and the database. This limits the total size of the dataset to the capacity of the file system where the external file resides and limits the data transfer to a single file write stream.
+
+More efficient data loading options for large datasets take advantage of the HAWQ MPP architecture, using the HAWQ segments to load data in parallel. These methods allow data to load simultaneously from multiple file systems, through multiple NICs, on multiple hosts, achieving very high data transfer rates. External tables allow you to access external files from within the database as if they are regular database tables. When used with `gpfdist`, the HAWQ parallel file distribution program, external tables provide full parallelism by using the resources of all HAWQ segments to load or unload data.
+
+HAWQ leverages the parallel architecture of the Hadoop Distributed File System to access files on that system.
+
+- **[Working with File-Based External Tables](../../datamgmt/load/g-working-with-file-based-ext-tables.html)**
+
+- **[Using the Greenplum Parallel File Server (gpfdist)](../../datamgmt/load/g-using-the-greenplum-parallel-file-server--gpfdist-.html)**
+
+- **[Creating and Using Web External Tables](../../datamgmt/load/g-creating-and-using-web-external-tables.html)**
+
+- **[Loading Data Using an External Table](../../datamgmt/load/g-loading-data-using-an-external-table.html)**
+
+- **[Loading and Writing Non-HDFS Custom Data](../../datamgmt/load/g-loading-and-writing-non-hdfs-custom-data.html)**
+
+- **[Creating External Tables - Examples](../../datamgmt/load/creating-external-tables-examples.html#topic44)**
+
+- **[Handling Load Errors](../../datamgmt/load/g-handling-load-errors.html)**
+
+- **[Loading Data with hawq load](../../datamgmt/load/g-loading-data-with-hawqload.html)**
+
+- **[Loading Data with COPY](../../datamgmt/load/g-loading-data-with-copy.html)**
+
+- **[Running COPY in Single Row Error Isolation Mode](../../datamgmt/load/g-running-copy-in-single-row-error-isolation-mode.html)**
+
+- **[Optimizing Data Load and Query Performance](../../datamgmt/load/g-optimizing-data-load-and-query-performance.html)**
+
+- **[Unloading Data from HAWQ](../../datamgmt/load/g-unloading-data-from-greenplum-database.html)**
+
+- **[Transforming XML Data](../../datamgmt/load/g-transforming-xml-data.html)**
+
+- **[Formatting Data Files](../../datamgmt/load/g-formatting-data-files.html)**
+
+
diff --git a/datamgmt/load/g-loading-and-writing-non-hdfs-custom-data.html.md.erb b/datamgmt/load/g-loading-and-writing-non-hdfs-custom-data.html.md.erb
new file mode 100644
index 0000000..e826963
--- /dev/null
+++ b/datamgmt/load/g-loading-and-writing-non-hdfs-custom-data.html.md.erb
@@ -0,0 +1,9 @@
+---
+title: Loading and Writing Non-HDFS Custom Data
+---
+
+HAWQ supports `TEXT` and `CSV` formats for importing and exporting data. You can load and write the data in other formats by defining and using a custom format or custom protocol.
+
+- **[Using a Custom Format](../../datamgmt/load/g-using-a-custom-format.html)**
+
+
diff --git a/datamgmt/load/g-loading-data-using-an-external-table.html.md.erb b/datamgmt/load/g-loading-data-using-an-external-table.html.md.erb
new file mode 100644
index 0000000..9af2b13
--- /dev/null
+++ b/datamgmt/load/g-loading-data-using-an-external-table.html.md.erb
@@ -0,0 +1,18 @@
+---
+title: Loading Data Using an External Table
+---
+
+Use SQL commands such as `INSERT` and `SELECT` to query a readable external table, the same way that you query a regular database table. For example, to load travel expense data from an external table, `ext_expenses`, into a database table,` expenses_travel`:
+
+``` sql
+=# INSERT INTO expenses_travel
+SELECT * from ext_expenses where category='travel';
+```
+
+To load all data into a new database table:
+
+``` sql
+=# CREATE TABLE expenses AS SELECT * from ext_expenses;
+```
+
+
diff --git a/datamgmt/load/g-loading-data-with-copy.html.md.erb b/datamgmt/load/g-loading-data-with-copy.html.md.erb
new file mode 100644
index 0000000..72e5ac6
--- /dev/null
+++ b/datamgmt/load/g-loading-data-with-copy.html.md.erb
@@ -0,0 +1,11 @@
+---
+title: Loading Data with COPY
+---
+
+`COPY FROM` copies data from a file or standard input into a table and appends the data to the table contents. `COPY` is non-parallel: data is loaded in a single process using the HAWQ master instance. Using `COPY` is only recommended for very small data files.
+
+The `COPY` source file must be accessible to the master host. Specify the `COPY` source file name relative to the master host location.
+
+HAWQ copies data from `STDIN` or `STDOUT` using the connection between the client and the master server.
+
+
diff --git a/datamgmt/load/g-loading-data-with-hawqload.html.md.erb b/datamgmt/load/g-loading-data-with-hawqload.html.md.erb
new file mode 100644
index 0000000..a214cf1
--- /dev/null
+++ b/datamgmt/load/g-loading-data-with-hawqload.html.md.erb
@@ -0,0 +1,56 @@
+---
+title: Loading Data with hawq load
+---
+
+The HAWQ `hawq load` utility loads data using readable external tables and the HAWQ parallel file server ( `gpfdist` or `gpfdists`). It handles parallel file-based external table setup and allows users to configure their data format, external table definition, and `gpfdist` or `gpfdists` setup in a single configuration file.
+
+## To use hawq load<a id="topic62__du168147"></a>
+
+1. Ensure that your environment is set up to run `hawq load`. Some dependent files from your HAWQ /> installation are required, such as `gpfdist` and Python, as well as network access to the HAWQ segment hosts.
+2. Create your load control file. This is a YAML-formatted file that specifies the HAWQ connection information, `gpfdist` configuration information, external table options, and data format.
+
+ For example:
+
+ ``` pre
+ ---
+ VERSION: 1.0.0.1
+ DATABASE: ops
+ USER: gpadmin
+ HOST: mdw-1
+ PORT: 5432
+ GPLOAD:
+ INPUT:
+ - SOURCE:
+ LOCAL_HOSTNAME:
+ - etl1-1
+ - etl1-2
+ - etl1-3
+ - etl1-4
+ PORT: 8081
+ FILE:
+ - /var/load/data/*
+ - COLUMNS:
+ - name: text
+ - amount: float4
+ - category: text
+ - description: text
+ - date: date
+ - FORMAT: text
+ - DELIMITER: '|'
+ - ERROR_LIMIT: 25
+ - ERROR_TABLE: payables.err_expenses
+ OUTPUT:
+ - TABLE: payables.expenses
+ - MODE: INSERT
+ SQL:
+ - BEFORE: "INSERT INTO audit VALUES('start', current_timestamp)"
+ - AFTER: "INSERT INTO audit VALUES('end', current_timestamp)"
+ ```
+
+3. Run `hawq load`, passing in the load control file. For example:
+
+ ``` shell
+ $ hawq load -f my_load.yml
+ ```
+
+
diff --git a/datamgmt/load/g-moving-data-between-tables.html.md.erb b/datamgmt/load/g-moving-data-between-tables.html.md.erb
new file mode 100644
index 0000000..2603ae4
--- /dev/null
+++ b/datamgmt/load/g-moving-data-between-tables.html.md.erb
@@ -0,0 +1,12 @@
+---
+title: Moving Data between Tables
+---
+
+You can use `CREATE TABLE AS` or `INSERT...SELECT` to load external and web external table data into another (non-external) database table, and the data will be loaded in parallel according to the external or web external table definition.
+
+If an external table file or web external table data source has an error, one of the following will happen, depending on the isolation mode used:
+
+- **Tables without error isolation mode**: any operation that reads from that table fails. Loading from external and web external tables without error isolation mode is an all or nothing operation.
+- **Tables with error isolation mode**: the entire file will be loaded, except for the problematic rows (subject to the configured REJECT\_LIMIT)
+
+
diff --git a/datamgmt/load/g-optimizing-data-load-and-query-performance.html.md.erb b/datamgmt/load/g-optimizing-data-load-and-query-performance.html.md.erb
new file mode 100644
index 0000000..203efc5
--- /dev/null
+++ b/datamgmt/load/g-optimizing-data-load-and-query-performance.html.md.erb
@@ -0,0 +1,10 @@
+---
+title: Optimizing Data Load and Query Performance
+---
+
+Use the following tips to help optimize your data load and subsequent query performance.
+
+- Run `ANALYZE` after loading data. If you significantly altered the data in a table, run `ANALYZE` or `VACUUM ANALYZE` to update table statistics for the query optimizer. Current statistics ensure that the optimizer makes the best decisions during query planning and avoids poor performance due to inaccurate or nonexistent statistics.
+- Run `VACUUM` after load errors. If the load operation does not run in single row error isolation mode, the operation stops at the first error. The target table contains the rows loaded before the error occurred. You cannot access these rows, but they occupy disk space. Use the `VACUUM` command to recover the wasted space.
+
+
diff --git a/datamgmt/load/g-representing-null-values.html.md.erb b/datamgmt/load/g-representing-null-values.html.md.erb
new file mode 100644
index 0000000..4d4ffdd
--- /dev/null
+++ b/datamgmt/load/g-representing-null-values.html.md.erb
@@ -0,0 +1,7 @@
+---
+title: Representing NULL Values
+---
+
+`NULL` represents an unknown piece of data in a column or field. Within your data files you can designate a string to represent null values. The default string is `\N` (backslash-N) in `TEXT` mode, or an empty value with no quotations in `CSV` mode. You can also declare a different string using the `NULL` clause of `COPY`, `CREATE EXTERNAL TABLE `or the `hawq load` control file when defining your data format. For example, you can use an empty string if you do not want to distinguish nulls from empty strings. When using the HAWQ loading tools, any data item that matches the designated null string is considered a null value.
+
+
diff --git a/datamgmt/load/g-running-copy-in-single-row-error-isolation-mode.html.md.erb b/datamgmt/load/g-running-copy-in-single-row-error-isolation-mode.html.md.erb
new file mode 100644
index 0000000..ba0603c
--- /dev/null
+++ b/datamgmt/load/g-running-copy-in-single-row-error-isolation-mode.html.md.erb
@@ -0,0 +1,17 @@
+---
+title: Running COPY in Single Row Error Isolation Mode
+---
+
+By default, `COPY` stops an operation at the first error: if the data contains an error, the operation fails and no data loads. If you run `COPY FROM` in *single row error isolation mode*, HAWQ skips rows that contain format errors and loads properly formatted rows. Single row error isolation mode applies only to rows in the input file that contain format errors. If the data contains a contraint error such as violation of a `NOT NULL` or `CHECK` constraint, the operation fails and no data loads.
+
+Specifying `SEGMENT REJECT LIMIT` runs the `COPY` operation in single row error isolation mode. Specify the acceptable number of error rows on each segment, after which the entire `COPY FROM` operation fails and no rows load. The error row count is for each HAWQ segment, not for the entire load operation.
+
+If the `COPY` operation does not reach the error limit, HAWQ loads all correctly-formatted rows and discards the error rows. The `LOG ERRORS INTO` clause allows you to keep error rows for further examination. Use `LOG ERRORS` to capture data formatting errors internally in HAWQ. For example:
+
+``` sql
+=> COPY country FROM '/data/gpdb/country_data'
+ WITH DELIMITER '|' LOG ERRORS INTO errtable
+ SEGMENT REJECT LIMIT 10 ROWS;
+```
+
+
diff --git a/datamgmt/load/g-starting-and-stopping-gpfdist.html.md.erb b/datamgmt/load/g-starting-and-stopping-gpfdist.html.md.erb
new file mode 100644
index 0000000..7e2cca9
--- /dev/null
+++ b/datamgmt/load/g-starting-and-stopping-gpfdist.html.md.erb
@@ -0,0 +1,42 @@
+---
+title: Starting and Stopping gpfdist
+---
+
+You can start `gpfdist` in your current directory location or in any directory that you specify. The default port is `8080`.
+
+From your current directory, type:
+
+``` shell
+$ gpfdist &
+```
+
+From a different directory, specify the directory from which to serve files, and optionally, the HTTP port to run on.
+
+To start `gpfdist` in the background and log output messages and errors to a log file:
+
+``` shell
+$ gpfdist -d /var/load_files -p 8081 -l /home/gpadmin/log &
+```
+
+For multiple `gpfdist` instances on the same ETL host (see [External Tables Using Multiple gpfdist Instances with Multiple NICs](g-about-gpfdist-setup-and-performance.html#topic14__du165882)), use a different base directory and port for each instance. For example:
+
+``` shell
+$ gpfdist -d /var/load_files1 -p 8081 -l /home/gpadmin/log1 &
+$ gpfdist -d /var/load_files2 -p 8082 -l /home/gpadmin/log2 &
+```
+
+To stop `gpfdist` when it is running in the background:
+
+First find its process id:
+
+``` shell
+$ ps -ef | grep gpfdist
+```
+
+Then kill the process, for example (where 3456 is the process ID in this example):
+
+``` shell
+$ kill 3456
+```
+
+
diff --git a/datamgmt/load/g-transfer-and-store-the-data.html.md.erb b/datamgmt/load/g-transfer-and-store-the-data.html.md.erb
new file mode 100644
index 0000000..8a6d7ab
--- /dev/null
+++ b/datamgmt/load/g-transfer-and-store-the-data.html.md.erb
@@ -0,0 +1,16 @@
+---
+title: Transfer and Store the Data
+---
+
+Use one of the following approaches to transform the data with `gpfdist`.
+
+- `GPLOAD` supports only input transformations, but is easier to implement in many cases.
+- `INSERT INTO SELECT FROM` supports both input and output transformations, but exposes more details.
+
+- **[Transforming with GPLOAD](../../datamgmt/load/g-transforming-with-gpload.html)**
+
+- **[Transforming with INSERT INTO SELECT FROM](../../datamgmt/load/g-transforming-with-insert-into-select-from.html)**
+
+- **[Configuration File Format](../../datamgmt/load/g-configuration-file-format.html)**
+
+
diff --git a/datamgmt/load/g-transforming-with-gpload.html.md.erb b/datamgmt/load/g-transforming-with-gpload.html.md.erb
new file mode 100644
index 0000000..438fedb
--- /dev/null
+++ b/datamgmt/load/g-transforming-with-gpload.html.md.erb
@@ -0,0 +1,30 @@
+---
+title: Transforming with GPLOAD
+---
+
+To transform data using the `GPLOAD ` control file, you must specify both the file name for the `TRANSFORM_CONFIG` file and the name of the `TRANSFORM` operation in the `INPUT` section of the `GPLOAD` control file.
+
+- `TRANSFORM_CONFIG `specifies the name of the `gpfdist` configuration file.
+- The `TRANSFORM` setting indicates the name of the transformation that is described in the file named in `TRANSFORM_CONFIG`.
+
+``` pre
+---
+VERSION: 1.0.0.1
+DATABASE: ops
+USER: gpadmin
+GPLOAD:
+INPUT:
+- TRANSFORM_CONFIG: config.yaml
+- TRANSFORM: prices_input
+- SOURCE:
+FILE: prices.xml
+```
+
+The transformation operation name must appear in two places: in the `TRANSFORM` setting of the `gpfdist` configuration file and in the `TRANSFORMATIONS` section of the file named in the `TRANSFORM_CONFIG` section.
+
+In the `GPLOAD` control file, the optional parameter `MAX_LINE_LENGTH` specifies the maximum length of a line in the XML transformation data that is passed to hawq load.
+
+The following diagram shows the relationships between the `GPLOAD` control file, the `gpfdist` configuration file, and the XML data file.
+
+<img src="../../images/03-gpload-files.jpg" class="image" width="415" height="258" />
+
diff --git a/datamgmt/load/g-transforming-with-insert-into-select-from.html.md.erb b/datamgmt/load/g-transforming-with-insert-into-select-from.html.md.erb
new file mode 100644
index 0000000..d91cc93
--- /dev/null
+++ b/datamgmt/load/g-transforming-with-insert-into-select-from.html.md.erb
@@ -0,0 +1,22 @@
+---
+title: Transforming with INSERT INTO SELECT FROM
+---
+
+Specify the transformation in the `CREATE EXTERNAL TABLE` definition's `LOCATION` clause. For example, the transform is shown in bold in the following command. (Run `gpfdist` first, using the command `gpfdist -c config.yaml`).
+
+``` sql
+CREATE READABLE EXTERNAL TABLE prices_readable (LIKE prices)
+ LOCATION ('gpfdist://hostname:8081/prices.xml#transform=prices_input')
+ FORMAT 'TEXT' (DELIMITER '|')
+ LOG ERRORS INTO error_log SEGMENT REJECT LIMIT 10;
+```
+
+In the command above, change *hostname* to your hostname. `prices_input` comes from the configuration file.
+
+The following query loads data into the `prices` table.
+
+``` sql
+INSERT INTO prices SELECT * FROM prices_readable;
+```
+
+
diff --git a/datamgmt/load/g-transforming-xml-data.html.md.erb b/datamgmt/load/g-transforming-xml-data.html.md.erb
new file mode 100644
index 0000000..f9520bb
--- /dev/null
+++ b/datamgmt/load/g-transforming-xml-data.html.md.erb
@@ -0,0 +1,34 @@
+---
+title: Transforming XML Data
+---
+
+The HAWQ data loader *gpfdist* provides transformation features to load XML data into a table and to write data from the HAWQ to XML files. The following diagram shows *gpfdist* performing an XML transform.
+
+<a id="topic75__du185408"></a>
+<span class="figtitleprefix">Figure: </span>External Tables using XML Transformations
+
+<img src="../../images/ext-tables-xml.png" class="image" />
+
+To load or extract XML data:
+
+- [Determine the Transformation Schema](g-determine-the-transformation-schema.html#topic76)
+- [Write a Transform](g-write-a-transform.html#topic77)
+- [Write the gpfdist Configuration](g-write-the-gpfdist-configuration.html#topic78)
+- [Load the Data](g-load-the-data.html#topic79)
+- [Transfer and Store the Data](g-transfer-and-store-the-data.html#topic80)
+
+The first three steps comprise most of the development effort. The last two steps are straightforward and repeatable, suitable for production.
+
+- **[Determine the Transformation Schema](../../datamgmt/load/g-determine-the-transformation-schema.html)**
+
+- **[Write a Transform](../../datamgmt/load/g-write-a-transform.html)**
+
+- **[Write the gpfdist Configuration](../../datamgmt/load/g-write-the-gpfdist-configuration.html)**
+
+- **[Load the Data](../../datamgmt/load/g-load-the-data.html)**
+
+- **[Transfer and Store the Data](../../datamgmt/load/g-transfer-and-store-the-data.html)**
+
+- **[XML Transformation Examples](../../datamgmt/load/g-xml-transformation-examples.html)**
+
+
diff --git a/datamgmt/load/g-troubleshooting-gpfdist.html.md.erb b/datamgmt/load/g-troubleshooting-gpfdist.html.md.erb
new file mode 100644
index 0000000..2e6a450
--- /dev/null
+++ b/datamgmt/load/g-troubleshooting-gpfdist.html.md.erb
@@ -0,0 +1,23 @@
+---
+title: Troubleshooting gpfdist
+---
+
+The segments access `gpfdist` at runtime. Ensure that the HAWQ segment hosts have network access to `gpfdist`. `gpfdist` is a web server: test connectivity by running the following command from each host in the HAWQ array (segments and master):
+
+``` shell
+$ wget http://gpfdist_hostname:port/filename
+```
+
+The `CREATE EXTERNAL TABLE` definition must have the correct host name, port, and file names for `gpfdist`. Specify file names and paths relative to the directory from which `gpfdist` serves files (the directory path specified when `gpfdist` started). See [Creating External Tables - Examples](creating-external-tables-examples.html#topic44).
+
+If you start `gpfdist` on your system and IPv6 networking is disabled, `gpfdist` displays this warning message when testing for an IPv6 port.
+
+``` pre
+[WRN gpfdist.c:2050] Creating the socket failed
+```
+
+If the corresponding IPv4 port is available, `gpfdist` uses that port and the warning for IPv6 port can be ignored. To see information about the ports that `gpfdist` tests, use the `-V` option.
+
+For information about IPv6 and IPv4 networking, see your operating system documentation.
+
+
diff --git a/datamgmt/load/g-unloading-data-from-greenplum-database.html.md.erb b/datamgmt/load/g-unloading-data-from-greenplum-database.html.md.erb
new file mode 100644
index 0000000..e0690ad
--- /dev/null
+++ b/datamgmt/load/g-unloading-data-from-greenplum-database.html.md.erb
@@ -0,0 +1,17 @@
+---
+title: Unloading Data from HAWQ
+---
+
+A writable external table allows you to select rows from other database tables and output the rows to files, named pipes, to applications, or as output targets for parallel MapReduce calculations. You can define file-based and web-based writable external tables.
+
+This topic describes how to unload data from HAWQ using parallel unload (writable external tables) and non-parallel unload (`COPY`).
+
+- **[Defining a File-Based Writable External Table](../../datamgmt/load/g-defining-a-file-based-writable-external-table.html)**
+
+- **[Defining a Command-Based Writable External Web Table](../../datamgmt/load/g-defining-a-command-based-writable-external-web-table.html)**
+
+- **[Unloading Data Using a Writable External Table](../../datamgmt/load/g-unloading-data-using-a-writable-external-table.html)**
+
+- **[Unloading Data Using COPY](../../datamgmt/load/g-unloading-data-using-copy.html)**
+
+
diff --git a/datamgmt/load/g-unloading-data-using-a-writable-external-table.html.md.erb b/datamgmt/load/g-unloading-data-using-a-writable-external-table.html.md.erb
new file mode 100644
index 0000000..377f2d6
--- /dev/null
+++ b/datamgmt/load/g-unloading-data-using-a-writable-external-table.html.md.erb
@@ -0,0 +1,17 @@
+---
+title: Unloading Data Using a Writable External Table
+---
+
+Writable external tables allow only `INSERT` operations. You must grant `INSERT` permission on a table to enable access to users who are not the table owner or a superuser. For example:
+
+``` sql
+GRANT INSERT ON writable_ext_table TO admin;
+```
+
+To unload data using a writable external table, select the data from the source table(s) and insert it into the writable external table. The resulting rows are output to the writable external table. For example:
+
+``` sql
+INSERT INTO writable_ext_table SELECT * FROM regular_table;
+```
+
+
diff --git a/datamgmt/load/g-unloading-data-using-copy.html.md.erb b/datamgmt/load/g-unloading-data-using-copy.html.md.erb
new file mode 100644
index 0000000..816a2b5
--- /dev/null
+++ b/datamgmt/load/g-unloading-data-using-copy.html.md.erb
@@ -0,0 +1,12 @@
+---
+title: Unloading Data Using COPY
+---
+
+`COPY TO` copies data from a table to a file (or standard input) on the HAWQ master host using a single process on the HAWQ master instance. Use `COPY` to output a table's entire contents, or filter the output using a `SELECT` statement. For example:
+
+``` sql
+COPY (SELECT * FROM country WHERE country_name LIKE 'A%')
+TO '/home/gpadmin/a_list_countries.out';
+```
+
+
diff --git a/datamgmt/load/g-url-based-web-external-tables.html.md.erb b/datamgmt/load/g-url-based-web-external-tables.html.md.erb
new file mode 100644
index 0000000..a115972
--- /dev/null
+++ b/datamgmt/load/g-url-based-web-external-tables.html.md.erb
@@ -0,0 +1,24 @@
+---
+title: URL-based Web External Tables
+---
+
+A URL-based web table accesses data from a web server using the HTTP protocol. Web table data is dynamic; the data is not rescannable.
+
+Specify the `LOCATION` of files on a web server using `http://`. The web data file(s) must reside on a web server that HAWQ segment hosts can access. The number of URLs specified corresponds to the minimum number of virtual segments that work in parallel to access the web table.
+
+The following sample command defines a web table that gets data from several URLs.
+
+``` sql
+=# CREATE EXTERNAL WEB TABLE ext_expenses (
+ name text, date date, amount float4, category text, description text)
+LOCATION ('http://intranet.company.com/expenses/sales/file.csv',
+ 'http://intranet.company.com/expenses/exec/file.csv',
+ 'http://intranet.company.com/expenses/finance/file.csv',
+ 'http://intranet.company.com/expenses/ops/file.csv',
+ 'http://intranet.company.com/expenses/marketing/file.csv',
+ 'http://intranet.company.com/expenses/eng/file.csv'
+ )
+FORMAT 'CSV' ( HEADER );
+```
+
+
diff --git a/datamgmt/load/g-using-a-custom-format.html.md.erb b/datamgmt/load/g-using-a-custom-format.html.md.erb
new file mode 100644
index 0000000..e83744a
--- /dev/null
+++ b/datamgmt/load/g-using-a-custom-format.html.md.erb
@@ -0,0 +1,23 @@
+---
+title: Using a Custom Format
+---
+
+You specify a custom data format in the `FORMAT` clause of `CREATE EXTERNAL TABLE`.
+
+```
+FORMAT 'CUSTOM' (formatter=format_function, key1=val1,...keyn=valn)
+```
+
+Where the `'CUSTOM'` keyword indicates that the data has a custom format and `formatter` specifies the function to use to format the data, followed by comma-separated parameters to the formatter function.
+
+HAWQ provides functions for formatting fixed-width data, but you must author the formatter functions for variable-width data. The steps are as follows.
+
+1. Author and compile input and output functions as a shared library.
+2. Specify the shared library function with `CREATE FUNCTION` in HAWQ.
+3. Use the `formatter` parameter of `CREATE EXTERNAL TABLE`'s `FORMAT` clause to call the function.
+
+- **[Importing and Exporting Fixed Width Data](../../datamgmt/load/g-importing-and-exporting-fixed-width-data.html)**
+
+- **[Examples - Read Fixed-Width Data](../../datamgmt/load/g-examples-read-fixed-width-data.html)**
+
+
diff --git a/datamgmt/load/g-using-the-greenplum-parallel-file-server--gpfdist-.html.md.erb b/datamgmt/load/g-using-the-greenplum-parallel-file-server--gpfdist-.html.md.erb
new file mode 100644
index 0000000..f357b2c
--- /dev/null
+++ b/datamgmt/load/g-using-the-greenplum-parallel-file-server--gpfdist-.html.md.erb
@@ -0,0 +1,19 @@
+---
+title: Using the Greenplum Parallel File Server (gpfdist)
+---
+
+The `gpfdist` protocol provides the best performance and is the easiest to set up. `gpfdist` ensures optimum use of all segments in your HAWQ system for external table reads.
+
+This topic describes the setup and management tasks for using `gpfdist` with external tables.
+
+- **[About gpfdist Setup and Performance](../../datamgmt/load/g-about-gpfdist-setup-and-performance.html)**
+
+- **[Controlling Segment Parallelism](../../datamgmt/load/g-controlling-segment-parallelism.html)**
+
+- **[Installing gpfdist](../../datamgmt/load/g-installing-gpfdist.html)**
+
+- **[Starting and Stopping gpfdist](../../datamgmt/load/g-starting-and-stopping-gpfdist.html)**
+
+- **[Troubleshooting gpfdist](../../datamgmt/load/g-troubleshooting-gpfdist.html)**
+
+
diff --git a/datamgmt/load/g-working-with-file-based-ext-tables.html.md.erb b/datamgmt/load/g-working-with-file-based-ext-tables.html.md.erb
new file mode 100644
index 0000000..e024a7d
--- /dev/null
+++ b/datamgmt/load/g-working-with-file-based-ext-tables.html.md.erb
@@ -0,0 +1,21 @@
+---
+title: Working with File-Based External Tables
+---
+
+External tables provide access to data stored in data sources outside of HAWQ as if the data were stored in regular database tables. Data can be read from or written to external tables.
+
+An external table is a HAWQ database table backed with data that resides outside of the database. An external table is either readable or writable. It can be used like a regular database table in SQL commands such as `SELECT` and `INSERT` and joined with other tables. External tables are most often used to load and unload database data.
+
+Web-based external tables provide access to data served by an HTTP server or an operating system process. See [Creating and Using Web External Tables](g-creating-and-using-web-external-tables.html#topic31) for more about web-based tables.
+
+- **[Accessing File-Based External Tables](../../datamgmt/load/g-external-tables.html)**
+
+ External tables enable accessing external files as if they are regular database tables. They are often used to move data into and out of a HAWQ database.
+
+- **[gpfdist Protocol](../../datamgmt/load/g-gpfdist-protocol.html)**
+
+- **[gpfdists Protocol](../../datamgmt/load/g-gpfdists-protocol.html)**
+
+- **[Handling Errors in External Table Data](../../datamgmt/load/g-handling-errors-ext-table-data.html)**
+
+
diff --git a/datamgmt/load/g-write-a-transform.html.md.erb b/datamgmt/load/g-write-a-transform.html.md.erb
new file mode 100644
index 0000000..6b35ab2
--- /dev/null
+++ b/datamgmt/load/g-write-a-transform.html.md.erb
@@ -0,0 +1,48 @@
+---
+title: Write a Transform
+---
+
+The transform specifies what to extract from the data.You can use any authoring environment and language appropriate for your project. For XML transformations, choose a technology such as XSLT, Joost (STX), Java, Python, or Perl, based on the goals and scope of the project.
+
+In the price example, the next step is to transform the XML data into a simple two-column delimited format.
+
+``` pre
+708421|19.99
+708466|59.25
+711121|24.99
+```
+
+The following STX transform, called *input\_transform.stx*, completes the data transformation.
+
+``` xml
+<?xml version="1.0"?>
+<stx:transform version="1.0"
+ xmlns:stx="http://stx.sourceforge.net/2002/ns"
+ pass-through="none">
+ <!-- declare variables -->
+ <stx:variable name="itemnumber"/>
+ <stx:variable name="price"/>
+ <!-- match and output prices as columns delimited by | -->
+ <stx:template match="/prices/pricerecord">
+ <stx:process-children/>
+ <stx:value-of select="$itemnumber"/>
+<stx:text>|</stx:text>
+ <stx:value-of select="$price"/> <stx:text>
+</stx:text>
+ </stx:template>
+ <stx:template match="itemnumber">
+ <stx:assign name="itemnumber" select="."/>
+ </stx:template>
+ <stx:template match="price">
+ <stx:assign name="price" select="."/>
+ </stx:template>
+</stx:transform>
+```
+
+This STX transform declares two temporary variables, `itemnumber` and `price`, and the following rules.
+
+1. When an element that satisfies the XPath expression `/prices/pricerecord` is found, examine the child elements and generate output that contains the value of the `itemnumber` variable, a `|` character, the value of the price variable, and a newline.
+2. When an `<itemnumber>` element is found, store the content of that element in the variable `itemnumber`.
+3. When a <price> element is found, store the content of that element in the variable `price`.
+
+
diff --git a/datamgmt/load/g-write-the-gpfdist-configuration.html.md.erb b/datamgmt/load/g-write-the-gpfdist-configuration.html.md.erb
new file mode 100644
index 0000000..89733cd
--- /dev/null
+++ b/datamgmt/load/g-write-the-gpfdist-configuration.html.md.erb
@@ -0,0 +1,61 @@
+---
+title: Write the gpfdist Configuration
+---
+
+The `gpfdist` configuration is specified as a YAML 1.1 document. It specifies rules that `gpfdist` uses to select a Transform to apply when loading or extracting data.
+
+This example `gpfdist` configuration contains the following items:
+
+- the `config.yaml` file defining `TRANSFORMATIONS`
+- the `input_transform.sh` wrapper script, referenced in the `config.yaml` file
+- the `input_transform.stx` joost transformation, called from `input_transform.sh`
+
+Aside from the ordinary YAML rules, such as starting the document with three dashes (`---`), a `gpfdist` configuration must conform to the following restrictions:
+
+1. a `VERSION` setting must be present with the value `1.0.0.1`.
+2. a `TRANSFORMATIONS` setting must be present and contain one or more mappings.
+3. Each mapping in the `TRANSFORMATION` must contain:
+ - a `TYPE` with the value 'input' or 'output'
+ - a `COMMAND` indicating how the transform is run.
+
+4. Each mapping in the `TRANSFORMATION` can contain optional `CONTENT`, `SAFE`, and `STDERR` settings.
+
+The following `gpfdist` configuration called `config.YAML` applies to the prices example. The initial indentation on each line is significant and reflects the hierarchical nature of the specification. The name `prices_input` in the following example will be referenced later when creating the table in SQL.
+
+``` pre
+---
+VERSION: 1.0.0.1
+TRANSFORMATIONS:
+ prices_input:
+ TYPE: input
+ COMMAND: /bin/bash input_transform.sh %filename%
+```
+
+The `COMMAND` setting uses a wrapper script called `input_transform.sh` with a `%filename%` placeholder. When `gpfdist` runs the `prices_input` transform, it invokes `input_transform.sh` with `/bin/bash` and replaces the `%filename%` placeholder with the path to the input file to transform. The wrapper script called `input_transform.sh` contains the logic to invoke the STX transformation and return the output.
+
+If Joost is used, the Joost STX engine must be installed.
+
+``` bash
+#!/bin/bash
+# input_transform.sh - sample input transformation,
+# demonstrating use of Java and Joost STX to convert XML into
+# text to load into HAWQ.
+# java arguments:
+# -jar joost.jar joost STX engine
+# -nodecl don't generate a <?xml?> declaration
+# $1 filename to process
+# input_transform.stx the STX transformation
+#
+# the AWK step eliminates a blank line joost emits at the end
+java \
+ -jar joost.jar \
+ -nodecl \
+ $1 \
+ input_transform.stx \
+ | awk 'NF>0
+```
+
+The `input_transform.sh` file uses the Joost STX engine with the AWK interpreter. The following diagram shows the process flow as `gpfdist` runs the transformation.
+
+<img src="../../images/02-pipeline.png" class="image" width="462" height="190" />
+
diff --git a/datamgmt/load/g-xml-transformation-examples.html.md.erb b/datamgmt/load/g-xml-transformation-examples.html.md.erb
new file mode 100644
index 0000000..12ad1d6
--- /dev/null
+++ b/datamgmt/load/g-xml-transformation-examples.html.md.erb
@@ -0,0 +1,13 @@
+---
+title: XML Transformation Examples
+---
+
+The following examples demonstrate the complete process for different types of XML data and STX transformations. Files and detailed instructions associated with these examples can be downloaded from the Apache site `gpfdist_transform` tools demo page. Read the README file before you run the examples.
+
+- **[Command-based Web External Tables](../../datamgmt/load/g-example-1-dblp-database-publications-in-demo-directory.html)**
+
+- **[Example using IRS MeF XML Files (In demo Directory)](../../datamgmt/load/g-example-irs-mef-xml-files-in-demo-directory.html)**
+
+- **[Example using WITSML™ Files (In demo Directory)](../../datamgmt/load/g-example-witsml-files-in-demo-directory.html)**
+
+
diff --git a/ddl/ddl-partition.md b/ddl/ddl-partition.md
index e3aaa02..367d010 100644
--- a/ddl/ddl-partition.md
+++ b/ddl/ddl-partition.md
@@ -37,7 +37,7 @@
Do not create more partitions than are needed. Creating too many partitions can slow down management and maintenance jobs, such as vacuuming, recovering segments, expanding the cluster, checking disk usage, and others.
-Partitioning does not improve query performance unless the query optimizer can eliminate partitions based on the query predicates. Queries that scan every partition run slower than if the table were not partitioned, so avoid partitioning if few of your queries achieve partition elimination. Check the explain plan for queries to make sure that partitions are eliminated. See [Query Profiling](/200/hawq/query/query-profiling.html) for more about partition elimination.
+Partitioning does not improve query performance unless the query optimizer can eliminate partitions based on the query predicates. Queries that scan every partition run slower than if the table were not partitioned, so avoid partitioning if few of your queries achieve partition elimination. Check the explain plan for queries to make sure that partitions are eliminated. See [Query Profiling](/20/query/query-profiling.html) for more about partition elimination.
Be very careful with multi-level partitioning because the number of partition files can grow very quickly. For example, if a table is partitioned by both day and city, and there are 1,000 days of data and 1,000 cities, the total number of partitions is one million. Column-oriented tables store each column in a physical table, so if this table has 100 columns, the system would be required to manage 100 million files for the table.
diff --git a/ddl/ddl-table.md b/ddl/ddl-table.md
index b74a1cb..7ad9ff4 100644
--- a/ddl/ddl-table.md
+++ b/ddl/ddl-table.md
@@ -126,7 +126,7 @@
`CREATE TABLE`'s optional clause `DISTRIBUTED BY` specifies the distribution policy for a table. The default is a random distribution policy. You can also choose to distribute data as a hash-based policy, where the `bucketnum` attribute sets the number of hash buckets used by a hash-distributed table. HASH distributed tables are created with the number of hash buckets specified by the `default_hash_table_bucket_number` parameter.
-Policies for different application scenarios can be specified to optimize performance. The number of virtual segments used for query execution can now be tuned using the `hawq_rm_nvseg_perquery_limit `and `hawq_rm_nvseg_perquery_perseg_limit` parameters, in connection with the `default_hash_table_bucket_number` parameter, which sets the default `bucketnum`. For more information, see the guidelines for Virtual Segments in the next section and in [Query Performance](/200/hawq/query/query-performance.html#topic38).
+Policies for different application scenarios can be specified to optimize performance. The number of virtual segments used for query execution can now be tuned using the `hawq_rm_nvseg_perquery_limit `and `hawq_rm_nvseg_perquery_perseg_limit` parameters, in connection with the `default_hash_table_bucket_number` parameter, which sets the default `bucketnum`. For more information, see the guidelines for Virtual Segments in the next section and in [Query Performance](/20/query/query-performance.html#topic38).
#### Performance Tuning <a id="topic_wff_mqm_gv"></a>
diff --git a/images/02-pipeline.png b/images/02-pipeline.png
new file mode 100644
index 0000000..26fec1b
--- /dev/null
+++ b/images/02-pipeline.png
Binary files differ
diff --git a/images/03-gpload-files.jpg b/images/03-gpload-files.jpg
new file mode 100644
index 0000000..d50435f
--- /dev/null
+++ b/images/03-gpload-files.jpg
Binary files differ
diff --git a/images/basic_query_flow.png b/images/basic_query_flow.png
new file mode 100644
index 0000000..59172a2
--- /dev/null
+++ b/images/basic_query_flow.png
Binary files differ
diff --git a/images/ext-tables-xml.png b/images/ext-tables-xml.png
new file mode 100644
index 0000000..f208828
--- /dev/null
+++ b/images/ext-tables-xml.png
Binary files differ
diff --git a/images/ext_tables.jpg b/images/ext_tables.jpg
new file mode 100644
index 0000000..d5a0940
--- /dev/null
+++ b/images/ext_tables.jpg
Binary files differ
diff --git a/images/ext_tables_multinic.jpg b/images/ext_tables_multinic.jpg
new file mode 100644
index 0000000..fcf09c4
--- /dev/null
+++ b/images/ext_tables_multinic.jpg
Binary files differ
diff --git a/images/gangs.jpg b/images/gangs.jpg
new file mode 100644
index 0000000..0d14585
--- /dev/null
+++ b/images/gangs.jpg
Binary files differ
diff --git a/images/gporca.png b/images/gporca.png
new file mode 100644
index 0000000..2909443
--- /dev/null
+++ b/images/gporca.png
Binary files differ
diff --git a/images/hawq_hcatalog.png b/images/hawq_hcatalog.png
new file mode 100644
index 0000000..35b74c3
--- /dev/null
+++ b/images/hawq_hcatalog.png
Binary files differ
diff --git a/images/slice_plan.jpg b/images/slice_plan.jpg
new file mode 100644
index 0000000..ad8da83
--- /dev/null
+++ b/images/slice_plan.jpg
Binary files differ
diff --git a/index.html.md.erb b/index.html.md.erb
new file mode 100644
index 0000000..336ca01
--- /dev/null
+++ b/index.html.md.erb
@@ -0,0 +1,602 @@
+- [Managing Data](datamgmt/dml.html)
+ - [Basic Data Operations](datamgmt/BasicDataOperations.html)
+ - [About Database Statistics](datamgmt/about_statistics.html)
+ - [Concurrency Control](datamgmt/ConcurrencyControl.html)
+ - [Working with Transactions](datamgmt/Transactions.html)
+ - [Loading and Unloading Data](datamgmt/load/g-loading-and-unloading-data.html)
+ - [Working with File-Based External Tables](datamgmt/load/g-working-with-file-based-ext-tables.html)
+ - [Accessing File-Based External Tables](datamgmt/load/g-external-tables.html)
+ - [gpfdist Protocol](datamgmt/load/g-gpfdist-protocol.html)
+ - [gpfdists Protocol](datamgmt/load/g-gpfdists-protocol.html)
+ - [Handling Errors in External Table Data](datamgmt/load/g-handling-errors-ext-table-data.html)
+ - [Using the Greenplum Parallel File Server (gpfdist)](datamgmt/load/g-using-the-greenplum-parallel-file-server--gpfdist-.html)
+ - [About gpfdist Setup and Performance](datamgmt/load/g-about-gpfdist-setup-and-performance.html)
+ - [Controlling Segment Parallelism](datamgmt/load/g-controlling-segment-parallelism.html)
+ - [Installing gpfdist](datamgmt/load/g-installing-gpfdist.html)
+ - [Starting and Stopping gpfdist](datamgmt/load/g-starting-and-stopping-gpfdist.html)
+ - [Troubleshooting gpfdist](datamgmt/load/g-troubleshooting-gpfdist.html)
+ - [Creating and Using Web External Tables](datamgmt/load/g-creating-and-using-web-external-tables.html)
+ - [Command-based Web External Tables](datamgmt/load/g-command-based-web-external-tables.html)
+ - [URL-based Web External Tables](datamgmt/load/g-url-based-web-external-tables.html)
+ - [Loading Data Using an External Table](datamgmt/load/g-loading-data-using-an-external-table.html)
+ - [Loading and Writing Non-HDFS Custom Data](datamgmt/load/g-loading-and-writing-non-hdfs-custom-data.html)
+ - [Using a Custom Format](datamgmt/load/g-using-a-custom-format.html)
+ - [Importing and Exporting Fixed Width Data](datamgmt/load/g-importing-and-exporting-fixed-width-data.html)
+ - [Examples - Read Fixed-Width Data](datamgmt/load/g-examples-read-fixed-width-data.html)
+ - [Creating External Tables - Examples](datamgmt/load/creating-external-tables-examples.html#topic44)
+ - [Example 1 - Single gpfdist instance on single-NIC machine](datamgmt/load/creating-external-tables-examples.html#topic14)
+ - [Example 2 - Multiple gpfdist instances](datamgmt/load/creating-external-tables-examples.html#topic46)
+ - [Example 3 - Multiple gpfdists instances](datamgmt/load/creating-external-tables-examples.html#topic47)
+ - [Example 4 - Single gpfdist instance with error logging](datamgmt/load/creating-external-tables-examples.html#topic48)
+ - [Example 5 - Readable Web External Table with Script](datamgmt/load/creating-external-tables-examples.html#topic51)
+ - [Example 6 - Writable External Table with gpfdist](datamgmt/load/creating-external-tables-examples.html#topic52)
+ - [Example 7 - Writable External Web Table with Script](datamgmt/load/creating-external-tables-examples.html#topic102)
+ - [Example 8 - Readable and Writable External Tables with XML Transformations](datamgmt/load/creating-external-tables-examples.html#topic54)
+ - [Handling Load Errors](datamgmt/load/g-handling-load-errors.html)
+ - [Define an External Table with Single Row Error Isolation](datamgmt/load/g-define-an-external-table-with-single-row-error-isolation.html)
+ - [Capture Row Formatting Errors and Declare a Reject Limit](datamgmt/load/g-create-an-error-table-and-declare-a-reject-limit.html)
+ - [Identifying Invalid CSV Files in Error Table Data](datamgmt/load/g-identifying-invalid-csv-files-in-error-table-data.html)
+ - [Moving Data between Tables](datamgmt/load/g-moving-data-between-tables.html)
+ - [Loading Data with hawq load](datamgmt/load/g-loading-data-with-hawqload.html)
+ - [Loading Data with COPY](datamgmt/load/g-loading-data-with-copy.html)
+ - [Running COPY in Single Row Error Isolation Mode](datamgmt/load/g-running-copy-in-single-row-error-isolation-mode.html)
+ - [Optimizing Data Load and Query Performance](datamgmt/load/g-optimizing-data-load-and-query-performance.html)
+ - [Unloading Data from HAWQ](datamgmt/load/g-unloading-data-from-greenplum-database.html)
+ - [Defining a File-Based Writable External Table](datamgmt/load/g-defining-a-file-based-writable-external-table.html)
+ - [Example - HAWQ file server (gpfdist)](datamgmt/load/g-example-greenplum-file-server-gpfdist.html)
+ - [Defining a Command-Based Writable External Web Table](datamgmt/load/g-defining-a-command-based-writable-external-web-table.html)
+ - [Disabling EXECUTE for Web or Writable External Tables](datamgmt/load/g-disabling-execute-for-web-or-writable-external-tables.html)
+ - [Unloading Data Using a Writable External Table](datamgmt/load/g-unloading-data-using-a-writable-external-table.html)
+ - [Unloading Data Using COPY](datamgmt/load/g-unloading-data-using-copy.html)
+ - [Transforming XML Data](datamgmt/load/g-transforming-xml-data.html)
+ - [Determine the Transformation Schema](datamgmt/load/g-determine-the-transformation-schema.html)
+ - [Write a Transform](datamgmt/load/g-write-a-transform.html)
+ - [Write the gpfdist Configuration](datamgmt/load/g-write-the-gpfdist-configuration.html)
+ - [Load the Data](datamgmt/load/g-load-the-data.html)
+ - [Transfer and Store the Data](datamgmt/load/g-transfer-and-store-the-data.html)
+ - [Transforming with GPLOAD](datamgmt/load/g-transforming-with-gpload.html)
+ - [Transforming with INSERT INTO SELECT FROM](datamgmt/load/g-transforming-with-insert-into-select-from.html)
+ - [Configuration File Format](datamgmt/load/g-configuration-file-format.html)
+ - [XML Transformation Examples](datamgmt/load/g-xml-transformation-examples.html)
+ - [Command-based Web External Tables](datamgmt/load/g-example-1-dblp-database-publications-in-demo-directory.html)
+ - [Example using IRS MeF XML Files (In demo Directory)](datamgmt/load/g-example-irs-mef-xml-files-in-demo-directory.html)
+ - [Example using WITSML™ Files (In demo Directory)](datamgmt/load/g-example-witsml-files-in-demo-directory.html)
+ - [Formatting Data Files](datamgmt/load/g-formatting-data-files.html)
+ - [Formatting Rows](datamgmt/load/g-formatting-rows.html)
+ - [Formatting Columns](datamgmt/load/g-formatting-columns.html)
+ - [Representing NULL Values](datamgmt/load/g-representing-null-values.html)
+ - [Escaping](datamgmt/load/g-escaping.html)
+ - [Escaping in Text Formatted Files](datamgmt/load/g-escaping-in-text-formatted-files.html)
+ - [Escaping in CSV Formatted Files](datamgmt/load/g-escaping-in-csv-formatted-files.html)
+ - [Character Encoding](datamgmt/load/g-character-encoding.html)
+ - [Working with PXF and External Data](pxf/PivotalExtensionFrameworkPXF.html)
+ - [Installing PXF Plug-ins](pxf/InstallPXFPlugins.html)
+ - [Installing the PXF HDFS Plug-in](pxf/InstallPXFPlugins.html#topic_qsz_wq5_25)
+ - [Installing the PXF Hive Plug-in](pxf/InstallPXFPlugins.html#topic_s1y_1v5_25)
+ - [Installing the PXF HBase Plug-in](pxf/InstallPXFPlugins.html#topic_std_cv5_25)
+ - [Installation and Log File Directories](pxf/InstallPXFPlugins.html#directories_and_logs)
+ - [Configuring PXF](pxf/ConfigurePXF.html)
+ - [Setting up the Java Classpath](pxf/ConfigurePXF.html#settingupthejavaclasspath)
+ - [Setting up the JVM Command Line Options for the PXF Service](pxf/ConfigurePXF.html#settingupthejvmcommandlineoptionsforpxfservice)
+ - [Using PXF on a Secure HDFS Cluster](pxf/ConfigurePXF.html#topic_i3f_hvm_ss)
+ - [Credentials for Remote Services](pxf/ConfigurePXF.html#credentialsforremoteservices)
+ - [Accessing HDFS File Data](pxf/HDFSFileDataPXF.html)
+ - [Prerequisites](pxf/HDFSFileDataPXF.html#installingthepxfhdfsplugin)
+ - [Syntax](pxf/HDFSFileDataPXF.html#syntax1)
+ - [Accessing Data on a High Availability HDFS Cluster](pxf/HDFSFileDataPXF.html#accessingdataonahighavailabilityhdfscluster)
+ - [Using a Record Key with Key-Value File Formats](pxf/HDFSFileDataPXF.html#recordkeyinkey-valuefileformats)
+ - [Working with Avro Files](pxf/HDFSFileDataPXF.html#topic_oy3_qwm_ss)
+ - [Accessing Hive Data](pxf/HivePXF.html)
+ - [Prerequisites](pxf/HivePXF.html#installingthepxfhiveplugin)
+ - [Hive Command Line](pxf/HivePXF.html#hivecommandline)
+ - [Using PXF Tables to Query Hive](pxf/HivePXF.html#topic_p2s_lvl_25)
+ - [Syntax Example](pxf/HivePXF.html#syntax2)
+ - [Hive Complex Types](pxf/HivePXF.html#topic_b4v_g3n_25)
+ - [Using PXF and HCatalog to Query Hive](pxf/HivePXF.html#hcatalog)
+ - [Usage](pxf/HivePXF.html#topic_j1l_y55_c5)
+ - [Limitations](pxf/HivePXF.html#topic_r5k_pst_25)
+ - [Partition Filtering](pxf/HivePXF.html#partitionfiltering)
+ - [Using PXF with Hive Default Partitions](pxf/HivePXF.html#topic_fdm_zrh_1s)
+ - [Accessing Hive Tables in Parquet Format](pxf/HivePXF.html#topic_dbb_nz3_ts)
+ - [Accessing HBase Data](pxf/HBasePXF.html)
+ - [Prerequisites](pxf/HBasePXF.html#installingthepxfhbaseplugin)
+ - [Syntax](pxf/HBasePXF.html#syntax3)
+ - [Column Mapping](pxf/HBasePXF.html#columnmapping)
+ - [Row Key](pxf/HBasePXF.html#rowkey)
+ - [Direct Mapping](pxf/HBasePXF.html#directmapping)
+ - [Indirect Mapping (via Lookup Table)](pxf/HBasePXF.html#indirectmappingvialookuptable)
+ - [Using Profiles to Read and Write Data](pxf/ReadWritePXF.html)
+ - [Built-In Profiles](pxf/ReadWritePXF.html#built-inprofiles)
+ - [Adding and Updating Profiles](pxf/ReadWritePXF.html#addingandupdatingprofiles)
+ - [Custom Profile Example](pxf/ReadWritePXF.html#customprofileexample)
+ - [PXF External Tables and API](pxf/PXFExternalTableandAPIReference.html)
+ - [Creating an External Table](pxf/PXFExternalTableandAPIReference.html#creatinganexternaltable)
+ - [About the Java Class Services and Formats](pxf/PXFExternalTableandAPIReference.html#aboutthejavaclassservicesandformats)
+ - [Fragmenter](pxf/PXFExternalTableandAPIReference.html#fragmenter)
+ - [Accessor](pxf/PXFExternalTableandAPIReference.html#accessor)
+ - [Resolver](pxf/PXFExternalTableandAPIReference.html#resolver)
+ - [About Custom Profiles](pxf/PXFExternalTableandAPIReference.html#aboutcustomprofiles)
+ - [About Query Filter Push-Down](pxf/PXFExternalTableandAPIReference.html#aboutqueryfilterpush-down)
+ - [Examples](pxf/PXFExternalTableandAPIReference.html#reference)
+ - [External Table Examples](pxf/PXFExternalTableandAPIReference.html#externaltableexamples)
+ - [Plug-in Examples](pxf/PXFExternalTableandAPIReference.html#pluginexamples)
+ - [Renamed Package Reference](pxf/PXFExternalTableandAPIReference.html#topic_b44_yw4_c5)
+ - [Troubleshooting PXF](pxf/TroubleshootingPXF.html)
+ - [HAWQ InputFormat for MapReduce](datamgmt/HAWQInputFormatforMapReduce.html)
+- [Querying Data](query/query.html)
+ - [About HAWQ Query Processing](query/HAWQQueryProcessing.html)
+ - [About GPORCA](query/gporca/query-gporca-optimizer.html)
+ - [Overview of GPORCA](query/gporca/query-gporca-overview.html)
+ - [GPORCA Features and Enhancements](query/gporca/query-gporca-features.html)
+ - [Enabling GPORCA](query/gporca/query-gporca-enable.html)
+ - [Considerations when Using GPORCA](query/gporca/query-gporca-notes.html)
+ - [Determining The Query Optimizer In Use](query/gporca/query-gporca-fallback.html)
+ - [Changed Behavior with GPORCA](query/gporca/query-gporca-changed.html)
+ - [GPORCA Limitations](query/gporca/query-gporca-limitations.html)
+ - [Defining Queries](query/defining-queries.html)
+ - [Using Functions and Operators](query/functions-operators.html)
+ - [Query Performance](query/query-performance.html)
+ - [Query Profiling](query/query-profiling.html)
+- [Best Practices](bestpractices/HAWQBestPracticesOverview.html)
+ - [HAWQ Best Practices](bestpractices/general_bestpractices.html)
+ - [Operating HAWQ](bestpractices/operating_hawq_bestpractices.html)
+ - [Securing HAWQ](bestpractices/secure_bestpractices.html)
+ - [Managing Resources](bestpractices/managing_resources_bestpractices.html)
+ - [Managing Data](bestpractices/managing_data_bestpractices.html)
+ - [Querying Data](bestpractices/querying_data_bestpractices.html)
+- [Troubleshooting](troubleshooting/Troubleshooting.html)
+ - [Query Performance Issues](troubleshooting/Troubleshooting.html#topic_dwd_rnx_15)
+ - [Rejection of Query Resource Requests](troubleshooting/Troubleshooting.html#topic_vm5_znx_15)
+ - [Queries Cancelled Due to High VMEM Usage](troubleshooting/Troubleshooting.html#topic_qq4_rkl_wv)
+ - [Segments Do Not Appear in gp\_segment\_configuration](troubleshooting/Troubleshooting.html#topic_hlj_zxx_15)
+ - [Handling Segment Resource Fragmentation](troubleshooting/Troubleshooting.html#topic_mdz_q2y_15)
+- [HAWQ Reference](reference/hawq-reference.html)
+ - [Server Configuration Parameter Reference](reference/HAWQSiteConfig.html)
+ - [About Server Configuration Parameters](reference/guc/guc_config.html)
+ - [Configuration Parameter Categories](reference/guc/guc_category-list.html)
+ - [Append-Only Table Parameters](reference/guc/guc_category-list.html#topic_hfd_1tl_zp)
+ - [Client Connection Default Parameters](reference/guc/guc_category-list.html#topic39)
+ - [Connection and Authentication Parameters](reference/guc/guc_category-list.html#topic12)
+ - [Database and Tablespace/Filespace Parameters](reference/guc/guc_category-list.html#topic47)
+ - [Error Reporting and Logging Parameters](reference/guc/guc_category-list.html#topic29)
+ - [External Table Parameters](reference/guc/guc_category-list.html#topic45)
+ - [GPORCA Parameters](reference/guc/guc_category-list.html#topic57)
+ - [HAWQ Array Configuration Parameters](reference/guc/guc_category-list.html#topic49)
+ - [HAWQ PL/Java Extension Parameters](reference/guc/guc_category-list.html#topic56)
+ - [HAWQ Resource Management Parameters](reference/guc/guc_category-list.html#hawq_resource_management)
+ - [Lock Management Parameters](reference/guc/guc_category-list.html#topic43)
+ - [Past PostgreSQL Version Compatibility Parameters](reference/guc/guc_category-list.html#topic48)
+ - [Query Tuning Parameters](reference/guc/guc_category-list.html#topic21)
+ - [Statistics Collection Parameters](reference/guc/guc_category-list.html#statistics_collection)
+ - [System Resource Consumption Parameters](reference/guc/guc_category-list.html#topic15)
+ - [Configuration Parameters](reference/guc/parameter_definitions.html)
+ - [add\_missing\_from](reference/guc/parameter_definitions.html#add_missing_from)
+ - [application\_name](reference/guc/parameter_definitions.html#application_name)
+ - [array\_nulls](reference/guc/parameter_definitions.html#array_nulls)
+ - [authentication\_timeout](reference/guc/parameter_definitions.html#authentication_timeout)
+ - [backslash\_quote](reference/guc/parameter_definitions.html#backslash_quote)
+ - [block\_size](reference/guc/parameter_definitions.html#block_size)
+ - [bonjour\_name](reference/guc/parameter_definitions.html#bonjour_name)
+ - [check\_function\_bodies](reference/guc/parameter_definitions.html#check_function_bodies)
+ - [client\_encoding](reference/guc/parameter_definitions.html#client_encoding)
+ - [client\_min\_messages](reference/guc/parameter_definitions.html#client_min_messages)
+ - [cpu\_index\_tuple\_cost](reference/guc/parameter_definitions.html#cpu_index_tuple_cost)
+ - [cpu\_operator\_cost](reference/guc/parameter_definitions.html#cpu_operator_cost)
+ - [cpu\_tuple\_cost](reference/guc/parameter_definitions.html#cpu_tuple_cost)
+ - [cursor\_tuple\_fraction](reference/guc/parameter_definitions.html#cursor_tuple_fraction)
+ - [custom\_variable\_classes](reference/guc/parameter_definitions.html#custom_variable_classes)
+ - [DateStyle](reference/guc/parameter_definitions.html#DateStyle)
+ - [db\_user\_namespace](reference/guc/parameter_definitions.html#db_user_namespace)
+ - [deadlock\_timeout](reference/guc/parameter_definitions.html#deadlock_timeout)
+ - [debug\_assertions](reference/guc/parameter_definitions.html#debug_assertions)
+ - [debug\_pretty\_print](reference/guc/parameter_definitions.html#debug_pretty_print)
+ - [debug\_print\_parse](reference/guc/parameter_definitions.html#debug_print_parse)
+ - [debug\_print\_plan](reference/guc/parameter_definitions.html#debug_print_plan)
+ - [debug\_print\_prelim\_plan](reference/guc/parameter_definitions.html#debug_print_prelim_plan)
+ - [debug\_print\_rewritten](reference/guc/parameter_definitions.html#debug_print_rewritten)
+ - [debug\_print\_slice\_table](reference/guc/parameter_definitions.html#debug_print_slice_table)
+ - [default\_hash\_table\_bucket\_number](reference/guc/parameter_definitions.html#topic_fqj_4fd_kv)
+ - [default\_statistics\_target](reference/guc/parameter_definitions.html#default_statistics_target)
+ - [default\_tablespace](reference/guc/parameter_definitions.html#default_tablespace)
+ - [default\_transaction\_isolation](reference/guc/parameter_definitions.html#default_transaction_isolation)
+ - [default\_transaction\_read\_only](reference/guc/parameter_definitions.html#default_transaction_read_only)
+ - [dfs\_url](reference/guc/parameter_definitions.html#dfs_url)
+ - [dynamic\_library\_path](reference/guc/parameter_definitions.html#dynamic_library_path)
+ - [effective\_cache\_size](reference/guc/parameter_definitions.html#effective_cache_size)
+ - [enable\_bitmapscan](reference/guc/parameter_definitions.html#enable_bitmapscan)
+ - [enable\_groupagg](reference/guc/parameter_definitions.html#enable_groupagg)
+ - [enable\_hashagg](reference/guc/parameter_definitions.html#enable_hashagg)
+ - [enable\_hashjoin](reference/guc/parameter_definitions.html#enable_hashjoin)
+ - [enable\_indexscan](reference/guc/parameter_definitions.html#enable_indexscan)
+ - [enable\_mergejoin](reference/guc/parameter_definitions.html#enable_mergejoin)
+ - [enable\_nestloop](reference/guc/parameter_definitions.html#enable_nestloop)
+ - [enable\_seqscan](reference/guc/parameter_definitions.html#enable_seqscan)
+ - [enable\_sort](reference/guc/parameter_definitions.html#enable_sort)
+ - [enable\_tidscan](reference/guc/parameter_definitions.html#enable_tidscan)
+ - [escape\_string\_warning](reference/guc/parameter_definitions.html#escape_string_warning)
+ - [explain\_pretty\_print](reference/guc/parameter_definitions.html#explain_pretty_print)
+ - [extra\_float\_digits](reference/guc/parameter_definitions.html#extra_float_digits)
+ - [from\_collapse\_limit](reference/guc/parameter_definitions.html#from_collapse_limit)
+ - [gp\_adjust\_selectivity\_for\_outerjoins](reference/guc/parameter_definitions.html#gp_adjust_selectivity_for_outerjoins)
+ - [gp\_analyze\_relative\_error](reference/guc/parameter_definitions.html#gp_analyze_relative_error)
+ - [gp\_autostats\_mode](reference/guc/parameter_definitions.html#gp_autostats_mode)
+ - [gp\_autostats\_on\_change\_threshhold](reference/guc/parameter_definitions.html#topic_imj_zhf_gw)
+ - [gp\_backup\_directIO](reference/guc/parameter_definitions.html#gp_backup_directIO)
+ - [gp\_backup\_directIO\_read\_chunk\_mb](reference/guc/parameter_definitions.html#gp_backup_directIO_read_chunk_mb)
+ - [gp\_cached\_segworkers\_threshold](reference/guc/parameter_definitions.html#gp_cached_segworkers_threshold)
+ - [gp\_command\_count](reference/guc/parameter_definitions.html#gp_command_count)
+ - [gp\_connections\_per\_thread](reference/guc/parameter_definitions.html#gp_connections_per_thread)
+ - [gp\_debug\_linger](reference/guc/parameter_definitions.html#gp_debug_linger)
+ - [gp\_dynamic\_partition\_pruning](reference/guc/parameter_definitions.html#gp_dynamic_partition_pruning)
+ - [gp\_enable\_agg\_distinct](reference/guc/parameter_definitions.html#gp_enable_agg_distinct)
+ - [gp\_enable\_agg\_distinct\_pruning](reference/guc/parameter_definitions.html#gp_enable_agg_distinct_pruning)
+ - [gp\_enable\_direct\_dispatch](reference/guc/parameter_definitions.html#gp_enable_direct_dispatch)
+ - [gp\_enable\_fallback\_plan](reference/guc/parameter_definitions.html#gp_enable_fallback_plan)
+ - [gp\_enable\_fast\_sri](reference/guc/parameter_definitions.html#gp_enable_fast_sri)
+ - [gp\_enable\_groupext\_distinct\_gather](reference/guc/parameter_definitions.html#gp_enable_groupext_distinct_gather)
+ - [gp\_enable\_groupext\_distinct\_pruning](reference/guc/parameter_definitions.html#gp_enable_groupext_distinct_pruning)
+ - [gp\_enable\_multiphase\_agg](reference/guc/parameter_definitions.html#gp_enable_multiphase_agg)
+ - [gp\_enable\_predicate\_propagation](reference/guc/parameter_definitions.html#gp_enable_predicate_propagation)
+ - [gp\_enable\_preunique](reference/guc/parameter_definitions.html#gp_enable_preunique)
+ - [gp\_enable\_sequential\_window\_plans](reference/guc/parameter_definitions.html#gp_enable_sequential_window_plans)
+ - [gp\_enable\_sort\_distinct](reference/guc/parameter_definitions.html#gp_enable_sort_distinct)
+ - [gp\_enable\_sort\_limit](reference/guc/parameter_definitions.html#gp_enable_sort_limit)
+ - [gp\_external\_enable\_exec](reference/guc/parameter_definitions.html#gp_external_enable_exec)
+ - [gp\_external\_grant\_privileges](reference/guc/parameter_definitions.html#gp_external_grant_privileges)
+ - [gp\_external\_max\_segs](reference/guc/parameter_definitions.html#gp_external_max_segs)
+ - [gp\_filerep\_tcp\_keepalives\_count](reference/guc/parameter_definitions.html#gp_filerep_tcp_keepalives_count)
+ - [gp\_filerep\_tcp\_keepalives\_idle](reference/guc/parameter_definitions.html#gp_filerep_tcp_keepalives_idle)
+ - [gp\_filerep\_tcp\_keepalives\_interval](reference/guc/parameter_definitions.html#gp_filerep_tcp_keepalives_interval)
+ - [gp\_hashjoin\_tuples\_per\_bucket](reference/guc/parameter_definitions.html#gp_hashjoin_tuples_per_bucket)
+ - [gp\_idf\_deduplicate](reference/guc/parameter_definitions.html#gp_idf_deduplicate)
+ - [gp\_interconnect\_fc\_method](reference/guc/parameter_definitions.html#gp_interconnect_fc_method)
+ - [gp\_interconnect\_hash\_multiplier](reference/guc/parameter_definitions.html#gp_interconnect_hash_multiplier)
+ - [gp\_interconnect\_queue\_depth](reference/guc/parameter_definitions.html#gp_interconnect_queue_depth)
+ - [gp\_interconnect\_setup\_timeout](reference/guc/parameter_definitions.html#gp_interconnect_setup_timeout)
+ - [gp\_interconnect\_snd\_queue\_depth](reference/guc/parameter_definitions.html#gp_interconnect_snd_queue_depth)
+ - [gp\_interconnect\_type](reference/guc/parameter_definitions.html#gp_interconnect_type)
+ - [gp\_log\_format](reference/guc/parameter_definitions.html#gp_log_format)
+ - [gp\_max\_csv\_line\_length](reference/guc/parameter_definitions.html#gp_max_csv_line_length)
+ - [gp\_max\_databases](reference/guc/parameter_definitions.html#gp_max_databases)
+ - [gp\_max\_filespaces](reference/guc/parameter_definitions.html#gp_max_filespaces)
+ - [gp\_max\_packet\_size](reference/guc/parameter_definitions.html#gp_max_packet_size)
+ - [gp\_max\_plan\_size](reference/guc/parameter_definitions.html#gp_max_plan_size)
+ - [gp\_max\_tablespaces](reference/guc/parameter_definitions.html#gp_max_tablespaces)
+ - [gp\_motion\_cost\_per\_row](reference/guc/parameter_definitions.html#gp_motion_cost_per_row)
+ - [gp\_reject\_percent\_threshold](reference/guc/parameter_definitions.html#gp_reject_percent_threshold)
+ - [gp\_reraise\_signal](reference/guc/parameter_definitions.html#gp_reraise_signal)
+ - [gp\_role](reference/guc/parameter_definitions.html#gp_role)
+ - [gp\_safefswritesize](reference/guc/parameter_definitions.html#gp_safefswritesize)
+ - [gp\_segment\_connect\_timeout](reference/guc/parameter_definitions.html#gp_segment_connect_timeout)
+ - [gp\_segments\_for\_planner](reference/guc/parameter_definitions.html#gp_segments_for_planner)
+ - [gp\_session\_id](reference/guc/parameter_definitions.html#gp_session_id)
+ - [gp\_set\_proc\_affinity](reference/guc/parameter_definitions.html#gp_set_proc_affinity)
+ - [gp\_set\_read\_only](reference/guc/parameter_definitions.html#gp_set_read_only)
+ - [gp\_statistics\_pullup\_from\_child\_partition](reference/guc/parameter_definitions.html#gp_statistics_pullup_from_child_partition)
+ - [gp\_statistics\_use\_fkeys](reference/guc/parameter_definitions.html#gp_statistics_use_fkeys)
+ - [gp\_vmem\_protect\_segworker\_cache\_limit](reference/guc/parameter_definitions.html#gp_vmem_protect_segworker_cache_limit)
+ - [gp\_workfile\_checksumming](reference/guc/parameter_definitions.html#gp_workfile_checksumming)
+ - [gp\_workfile\_compress\_algorithm](reference/guc/parameter_definitions.html#gp_workfile_compress_algorithm)
+ - [gp\_workfile\_limit\_files\_per\_query](reference/guc/parameter_definitions.html#gp_workfile_limit_files_per_query)
+ - [gp\_workfile\_limit\_per\_query](reference/guc/parameter_definitions.html#gp_workfile_limit_per_query)
+ - [gp\_workfile\_limit\_per\_segment](reference/guc/parameter_definitions.html#gp_workfile_limit_per_segment)
+ - [hawq\_dfs\_url](reference/guc/parameter_definitions.html#hawq_dfs_url)
+ - [hawq\_global\_rm\_type](reference/guc/parameter_definitions.html#hawq_global_rm_type)
+ - [hawq\_master\_address\_host](reference/guc/parameter_definitions.html#hawq_master_address_host)
+ - [hawq\_master\_address\_port](reference/guc/parameter_definitions.html#hawq_master_address_port)
+ - [hawq\_master\_directory](reference/guc/parameter_definitions.html#hawq_master_directory)
+ - [hawq\_master\_temp\_directory](reference/guc/parameter_definitions.html#hawq_master_temp_directory)
+ - [hawq\_re\_memory\_overcommit\_max](reference/guc/parameter_definitions.html#hawq_re_memory_overcommit_max)
+ - [hawq\_rm\_cluster\_report\_period](reference/guc/parameter_definitions.html#hawq_rm_cluster_report)
+ - [hawq\_rm\_force\_alterqueue\_cancel\_queued\_request](reference/guc/parameter_definitions.html#hawq_rm_force_alterqueue_cancel_queued_request)
+ - [hawq\_rm\_master\_port](reference/guc/parameter_definitions.html#hawq_rm_master_port)
+ - [hawq\_rm\_memory\_limit\_perseg](reference/guc/parameter_definitions.html#hawq_rm_memory_limit_perseg)
+ - [hawq\_rm\_min\_resource\_perseg](reference/guc/parameter_definitions.html#hawq_rm_min_resource_perseg)
+ - [hawq\_rm\_nresqueue\_limit](reference/guc/parameter_definitions.html#hawq_rm_nresqueue_limit)
+ - [hawq\_rm\_nslice\_perseg\_limit](reference/guc/parameter_definitions.html#hawq_rm_nslice_perseg_limit)
+ - [hawq\_rm\_nvcore\_limit\_perseg](reference/guc/parameter_definitions.html#hawq_rm_nvcore_limit_perseg)
+ - [hawq\_rm\_nvseg\_perquery\_limit](reference/guc/parameter_definitions.html#hawq_rm_nvseg_perquery_limit)
+ - [hawq\_rm\_nvseg\_perquery\_perseg\_limit](reference/guc/parameter_definitions.html#hawq_rm_nvseg_perquery_perseg_limit)
+ - [hawq\_rm\_nvseg\_variance\_amon\_seg\_limit](reference/guc/parameter_definitions.html#hawq_rm_nvseg_variance_amon_seg_limit)
+ - [hawq\_rm\_rejectrequest\_nseg\_limit](reference/guc/parameter_definitions.html#hawq_rm_rejectrequest_nseg_limit)
+ - [hawq\_rm\_resource\_idle\_timeout](reference/guc/parameter_definitions.html#hawq_rm_resource_idle_timeout)
+ - [hawq\_rm\_return\_percent\_on\_overcommit](reference/guc/parameter_definitions.html#hawq_rm_return_percent_on_overcommit)
+ - [hawq\_rm\_segment\_heartbeat\_interval](reference/guc/parameter_definitions.html#hawq_rm_segment_heartbeat_interval)
+ - [hawq\_rm\_segment\_port](reference/guc/parameter_definitions.html#hawq_rm_segment_port)
+ - [hawq\_rm\_stmt\_nvseg](reference/guc/parameter_definitions.html#hawq_rm_stmt_nvseg)
+ - [hawq\_rm\_stmt\_vseg\_memory](reference/guc/parameter_definitions.html#hawq_rm_stmt_vseg_memory)
+ - [hawq\_rm\_tolerate\_nseg\_limit](reference/guc/parameter_definitions.html#hawq_rm_tolerate_nseg_limit)
+ - [hawq\_rm\_yarn\_address](reference/guc/parameter_definitions.html#hawq_rm_yarn_address)
+ - [hawq\_rm\_yarn\_app\_name](reference/guc/parameter_definitions.html#hawq_rm_yarn_app_name)
+ - [hawq\_rm\_yarn\_queue\_name](reference/guc/parameter_definitions.html#hawq_rm_yarn_queue_name)
+ - [hawq\_rm\_yarn\_scheduler\_address](reference/guc/parameter_definitions.html#hawq_rm_yarn_scheduler_address)
+ - [hawq\_segment\_address\_port](reference/guc/parameter_definitions.html#hawq_segment_address_port)
+ - [hawq\_segment\_directory](reference/guc/parameter_definitions.html#hawq_segment_directory)
+ - [hawq\_segment\_temp\_directory](reference/guc/parameter_definitions.html#hawq_segment_temp_directory)
+ - [integer\_datetimes](reference/guc/parameter_definitions.html#integer_datetimes)
+ - [IntervalStyle](reference/guc/parameter_definitions.html#IntervalStyle)
+ - [join\_collapse\_limit](reference/guc/parameter_definitions.html#join_collapse_limit)
+ - [krb\_caseins\_users](reference/guc/parameter_definitions.html#krb_caseins_users)
+ - [krb\_server\_keyfile](reference/guc/parameter_definitions.html#krb_server_keyfile)
+ - [krb\_srvname](reference/guc/parameter_definitions.html#krb_srvname)
+ - [lc\_collate](reference/guc/parameter_definitions.html#lc_collate)
+ - [lc\_ctype](reference/guc/parameter_definitions.html#lc_ctype)
+ - [lc\_messages](reference/guc/parameter_definitions.html#lc_messages)
+ - [lc\_monetary](reference/guc/parameter_definitions.html#lc_monetary)
+ - [lc\_numeric](reference/guc/parameter_definitions.html#lc_numeric)
+ - [lc\_time](reference/guc/parameter_definitions.html#lc_time)
+ - [listen\_addresses](reference/guc/parameter_definitions.html#listen_addresses)
+ - [local\_preload\_libraries](reference/guc/parameter_definitions.html#local_preload_libraries)
+ - [log\_autostats](reference/guc/parameter_definitions.html#log_autostats)
+ - [log\_connections](reference/guc/parameter_definitions.html#log_connections)
+ - [log\_disconnections](reference/guc/parameter_definitions.html#log_disconnections)
+ - [log\_dispatch\_stats](reference/guc/parameter_definitions.html#log_dispatch_stats)
+ - [log\_duration](reference/guc/parameter_definitions.html#log_duration)
+ - [log\_error\_verbosity](reference/guc/parameter_definitions.html#log_error_verbosity)
+ - [log\_executor\_stats](reference/guc/parameter_definitions.html#log_executor_stats)
+ - [log\_hostname](reference/guc/parameter_definitions.html#log_hostname)
+ - [log\_min\_duration\_statement](reference/guc/parameter_definitions.html#log_min_duration_statement)
+ - [log\_min\_error\_statement](reference/guc/parameter_definitions.html#log_min_error_statement)
+ - [log\_min\_messages](reference/guc/parameter_definitions.html#log_min_messages)
+ - [log\_parser\_stats](reference/guc/parameter_definitions.html#log_parser_stats)
+ - [log\_planner\_stats](reference/guc/parameter_definitions.html#log_planner_stats)
+ - [log\_rotation\_age](reference/guc/parameter_definitions.html#log_rotation_age)
+ - [log\_rotation\_size](reference/guc/parameter_definitions.html#log_rotation_size)
+ - [log\_statement](reference/guc/parameter_definitions.html#log_statement)
+ - [log\_statement\_stats](reference/guc/parameter_definitions.html#log_statement_stats)
+ - [log\_timezone](reference/guc/parameter_definitions.html#log_timezone)
+ - [log\_truncate\_on\_rotation](reference/guc/parameter_definitions.html#log_truncate_on_rotation)
+ - [max\_appendonly\_tables](reference/guc/parameter_definitions.html#max_appendonly_tables)
+ - [max\_connections](reference/guc/parameter_definitions.html#max_connections)
+ - [max\_files\_per\_process](reference/guc/parameter_definitions.html#max_files_per_process)
+ - [max\_fsm\_pages](reference/guc/parameter_definitions.html#max_fsm_pages)
+ - [max\_fsm\_relations](reference/guc/parameter_definitions.html#max_fsm_relations)
+ - [max\_function\_args](reference/guc/parameter_definitions.html#max_function_args)
+ - [max\_identifier\_length](reference/guc/parameter_definitions.html#max_identifier_length)
+ - [max\_index\_keys](reference/guc/parameter_definitions.html#max_index_keys)
+ - [max\_locks\_per\_transaction](reference/guc/parameter_definitions.html#max_locks_per_transaction)
+ - [max\_prepared\_transactions](reference/guc/parameter_definitions.html#max_prepared_transactions)
+ - [max\_stack\_depth](reference/guc/parameter_definitions.html#max_stack_depth)
+ - [optimizer](reference/guc/parameter_definitions.html#optimizer)
+ - [optimizer\_analyze\_root\_partition](reference/guc/parameter_definitions.html#optimizer_analyze_root_partition)
+ - [optimizer\_minidump](reference/guc/parameter_definitions.html#optimizer_minidump)
+ - [optimizer\_parts\_to\_force\_sort\_on\_insert](reference/guc/parameter_definitions.html#optimizer_parts_to_force_sort_on_insert)
+ - [optimizer\_prefer\_scalar\_dqa\_multistage\_agg](reference/guc/parameter_definitions.html#optimizer_prefer_scalar_dqa_multistage_agg)
+ - [password\_encryption](reference/guc/parameter_definitions.html#password_encryption)
+ - [pgstat\_track\_activity\_query\_size](reference/guc/parameter_definitions.html#pgstat_track_activity_query_size)
+ - [pljava\_classpath](reference/guc/parameter_definitions.html#pljava_classpath)
+ - [pljava\_statement\_cache\_size](reference/guc/parameter_definitions.html#pljava_statement_cache_size)
+ - [pljava\_release\_lingering\_savepoints](reference/guc/parameter_definitions.html#pljava_release_lingering_savepoints)
+ - [pljava\_vmoptions](reference/guc/parameter_definitions.html#pljava_vmoptions)
+ - [port](reference/guc/parameter_definitions.html#port)
+ - [pxf\_enable\_stat\_collection](reference/guc/parameter_definitions.html#pxf_enable_stat_collection)
+ - [pxf\_service\_address](reference/guc/parameter_definitions.html#pxf_service_address)
+ - [pxf\_service\_port](reference/guc/parameter_definitions.html#pxf_service_port)
+ - [pxf\_stat\_max\_fragments](reference/guc/parameter_definitions.html#pxf_stat_max_fragments)
+ - [random\_page\_cost](reference/guc/parameter_definitions.html#random_page_cost)
+ - [regex\_flavor](reference/guc/parameter_definitions.html#regex_flavor)
+ - [runaway\_detector\_activation\_percent](reference/guc/parameter_definitions.html#runaway_detector_activation_percent)
+ - [search\_path](reference/guc/parameter_definitions.html#search_path)
+ - [seg\_max\_connections](reference/guc/parameter_definitions.html#seg_max_connections)
+ - [seq\_page\_cost](reference/guc/parameter_definitions.html#seq_page_cost)
+ - [server\_encoding](reference/guc/parameter_definitions.html#server_encoding)
+ - [server\_version](reference/guc/parameter_definitions.html#server_version)
+ - [server\_version\_num](reference/guc/parameter_definitions.html#server_version_num)
+ - [shared\_buffers](reference/guc/parameter_definitions.html#shared_buffers)
+ - [shared\_preload\_libraries](reference/guc/parameter_definitions.html#shared_preload_libraries)
+ - [ssl](reference/guc/parameter_definitions.html#ssl)
+ - [ssl\_ciphers](reference/guc/parameter_definitions.html#ssl_ciphers)
+ - [standard\_conforming\_strings](reference/guc/parameter_definitions.html#standard_conforming_strings)
+ - [statement\_timeout](reference/guc/parameter_definitions.html#statement_timeout)
+ - [superuser\_reserved\_connections](reference/guc/parameter_definitions.html#superuser_reserved_connections)
+ - [tcp\_keepalives\_count](reference/guc/parameter_definitions.html#tcp_keepalives_count)
+ - [tcp\_keepalives\_idle](reference/guc/parameter_definitions.html#tcp_keepalives_idle)
+ - [tcp\_keepalives\_interval](reference/guc/parameter_definitions.html#tcp_keepalives_interval)
+ - [temp\_buffers](reference/guc/parameter_definitions.html#temp_buffers)
+ - [TimeZone](reference/guc/parameter_definitions.html#TimeZone)
+ - [timezone\_abbreviations](reference/guc/parameter_definitions.html#timezone_abbreviations)
+ - [track\_activities](reference/guc/parameter_definitions.html#track_activities)
+ - [track\_counts](reference/guc/parameter_definitions.html#track_counts)
+ - [transaction\_isolation](reference/guc/parameter_definitions.html#transaction_isolation)
+ - [transaction\_read\_only](reference/guc/parameter_definitions.html#transaction_read_only)
+ - [transform\_null\_equals](reference/guc/parameter_definitions.html#transform_null_equals)
+ - [unix\_socket\_directory](reference/guc/parameter_definitions.html#unix_socket_directory)
+ - [unix\_socket\_group](reference/guc/parameter_definitions.html#unix_socket_group)
+ - [unix\_socket\_permissions](reference/guc/parameter_definitions.html#unix_socket_permissions)
+ - [update\_process\_title](reference/guc/parameter_definitions.html#update_process_title)
+ - [vacuum\_cost\_delay](reference/guc/parameter_definitions.html#vacuum_cost_delay)
+ - [vacuum\_cost\_limit](reference/guc/parameter_definitions.html#vacuum_cost_limit)
+ - [vacuum\_cost\_page\_dirty](reference/guc/parameter_definitions.html#vacuum_cost_page_dirty)
+ - [vacuum\_cost\_page\_miss](reference/guc/parameter_definitions.html#vacuum_cost_page_miss)
+ - [vacuum\_freeze\_min\_age](reference/guc/parameter_definitions.html#vacuum_freeze_min_age)
+ - [xid\_stop\_limit](reference/guc/parameter_definitions.html#xid_stop_limit)
+ - [Sample hawq-site.xml Configuration File](reference/HAWQSampleSiteConfig.html)
+ - [HDFS Configuration Reference](reference/HDFSConfigurationParameterReference.html)
+ - [Environment Variables](reference/HAWQEnvironmentVariables.html)
+ - [Character Set Support Reference](reference/CharacterSetSupportReference.html)
+ - [Data Types](reference/HAWQDataTypes.html)
+ - [SQL Commands](reference/SQLCommandReference.html)
+ - [ABORT](reference/sql/ABORT.html)
+ - [ALTER AGGREGATE](reference/sql/ALTER-AGGREGATE.html)
+ - [ALTER DATABASE](reference/sql/ALTER-DATABASE.html)
+ - [ALTER FUNCTION](reference/sql/ALTER-FUNCTION.html)
+ - [ALTER OPERATOR](reference/sql/ALTER-OPERATOR.html)
+ - [ALTER OPERATOR CLASS](reference/sql/ALTER-OPERATOR-CLASS.html)
+ - [ALTER RESOURCE QUEUE](reference/sql/ALTER-RESOURCE-QUEUE.html)
+ - [ALTER ROLE](reference/sql/ALTER-ROLE.html)
+ - [ALTER TABLE](reference/sql/ALTER-TABLE.html)
+ - [ALTER TABLESPACE](reference/sql/ALTER-TABLESPACE.html)
+ - [ALTER TYPE](reference/sql/ALTER-TYPE.html)
+ - [ALTER USER](reference/sql/ALTER-USER.html)
+ - [ANALYZE](reference/sql/ANALYZE.html)
+ - [BEGIN](reference/sql/BEGIN.html)
+ - [CHECKPOINT](reference/sql/CHECKPOINT.html)
+ - [CLOSE](reference/sql/CLOSE.html)
+ - [COMMIT](reference/sql/COMMIT.html)
+ - [COPY](reference/sql/COPY.html)
+ - [CREATE AGGREGATE](reference/sql/CREATE-AGGREGATE.html)
+ - [CREATE DATABASE](reference/sql/CREATE-DATABASE.html)
+ - [CREATE EXTERNAL TABLE](reference/sql/CREATE-EXTERNAL-TABLE.html)
+ - [CREATE FUNCTION](reference/sql/CREATE-FUNCTION.html)
+ - [CREATE GROUP](reference/sql/CREATE-GROUP.html)
+ - [CREATE LANGUAGE](reference/sql/CREATE-LANGUAGE.html)
+ - [CREATE OPERATOR](reference/sql/CREATE-OPERATOR.html)
+ - [CREATE OPERATOR CLASS](reference/sql/CREATE-OPERATOR-CLASS.html)
+ - [CREATE RESOURCE QUEUE](reference/sql/CREATE-RESOURCE-QUEUE.html)
+ - [CREATE ROLE](reference/sql/CREATE-ROLE.html)
+ - [CREATE SCHEMA](reference/sql/CREATE-SCHEMA.html)
+ - [CREATE SEQUENCE](reference/sql/CREATE-SEQUENCE.html)
+ - [CREATE TABLE](reference/sql/CREATE-TABLE.html)
+ - [CREATE TABLE AS](reference/sql/CREATE-TABLE-AS.html)
+ - [CREATE TABLESPACE](reference/sql/CREATE-TABLESPACE.html)
+ - [CREATE TYPE](reference/sql/CREATE-TYPE.html)
+ - [CREATE USER](reference/sql/CREATE-USER.html)
+ - [CREATE VIEW](reference/sql/CREATE-VIEW.html)
+ - [DEALLOCATE](reference/sql/DEALLOCATE.html)
+ - [DECLARE](reference/sql/DECLARE.html)
+ - [DROP AGGREGATE](reference/sql/DROP-AGGREGATE.html)
+ - [DROP DATABASE](reference/sql/DROP-DATABASE.html)
+ - [DROP EXTERNAL TABLE](reference/sql/DROP-EXTERNAL-TABLE.html)
+ - [DROP FILESPACE](reference/sql/DROP-FILESPACE.html)
+ - [DROP FUNCTION](reference/sql/DROP-FUNCTION.html)
+ - [DROP GROUP](reference/sql/DROP-GROUP.html)
+ - [DROP OPERATOR](reference/sql/DROP-OPERATOR.html)
+ - [DROP OPERATOR CLASS](reference/sql/DROP-OPERATOR-CLASS.html)
+ - [DROP OWNED](reference/sql/DROP-OWNED.html)
+ - [DROP RESOURCE QUEUE](reference/sql/DROP-RESOURCE-QUEUE.html)
+ - [DROP ROLE](reference/sql/DROP-ROLE.html)
+ - [DROP SCHEMA](reference/sql/DROP-SCHEMA.html)
+ - [DROP SEQUENCE](reference/sql/DROP-SEQUENCE.html)
+ - [DROP TABLE](reference/sql/DROP-TABLE.html)
+ - [DROP TABLESPACE](reference/sql/DROP-TABLESPACE.html)
+ - [DROP TYPE](reference/sql/DROP-TYPE.html)
+ - [DROP USER](reference/sql/DROP-USER.html)
+ - [DROP VIEW](reference/sql/DROP-VIEW.html)
+ - [END](reference/sql/END.html)
+ - [EXECUTE](reference/sql/EXECUTE.html)
+ - [EXPLAIN](reference/sql/EXPLAIN.html)
+ - [FETCH](reference/sql/FETCH.html)
+ - [GRANT](reference/sql/GRANT.html)
+ - [INSERT](reference/sql/INSERT.html)
+ - [PREPARE](reference/sql/PREPARE.html)
+ - [REASSIGN OWNED](reference/sql/REASSIGN-OWNED.html)
+ - [RELEASE SAVEPOINT](reference/sql/RELEASE-SAVEPOINT.html)
+ - [RESET](reference/sql/RESET.html)
+ - [REVOKE](reference/sql/REVOKE.html)
+ - [ROLLBACK](reference/sql/ROLLBACK.html)
+ - [ROLLBACK TO SAVEPOINT](reference/sql/ROLLBACK-TO-SAVEPOINT.html)
+ - [SAVEPOINT](reference/sql/SAVEPOINT.html)
+ - [SELECT](reference/sql/SELECT.html)
+ - [SELECT INTO](reference/sql/SELECT-INTO.html)
+ - [SET](reference/sql/SET.html)
+ - [SET ROLE](reference/sql/SET-ROLE.html)
+ - [SET SESSION AUTHORIZATION](reference/sql/SET-SESSION-AUTHORIZATION.html)
+ - [SHOW](reference/sql/SHOW.html)
+ - [TRUNCATE](reference/sql/TRUNCATE.html)
+ - [VACUUM](reference/sql/VACUUM.html)
+ - [System Catalog Reference](reference/catalog/catalog_ref.html)
+ - [System Tables](reference/catalog/catalog_ref-tables.html)
+ - [System Views](reference/catalog/catalog_ref-views.html)
+ - [System Catalogs Definitions](reference/catalog/catalog_ref-html.html)
+ - [gp\_configuration\_history](reference/catalog/gp_configuration_history.html)
+ - [gp\_distribution\_policy](reference/catalog/gp_distribution_policy.html)
+ - [gp\_global\_sequence](reference/catalog/gp_global_sequence.html)
+ - [gp\_master\_mirroring](reference/catalog/gp_master_mirroring.html)
+ - [gp\_persistent\_database\_node](reference/catalog/gp_persistent_database_node.html)
+ - [gp\_persistent\_filespace\_node](reference/catalog/gp_persistent_filespace_node.html)
+ - [gp\_persistent\_relation\_node](reference/catalog/gp_persistent_relation_node.html)
+ - [gp\_persistent\_relfile\_node](reference/catalog/gp_persistent_relfile_node.html)
+ - [gp\_persistent\_tablespace\_node](reference/catalog/gp_persistent_tablespace_node.html)
+ - [gp\_relfile\_node](reference/catalog/gp_relfile_node.html)
+ - [gp\_segment\_configuration](reference/catalog/gp_segment_configuration.html)
+ - [gp\_version\_at\_initdb](reference/catalog/gp_version_at_initdb.html)
+ - [pg\_aggregate](reference/catalog/pg_aggregate.html)
+ - [pg\_am](reference/catalog/pg_am.html)
+ - [pg\_amop](reference/catalog/pg_amop.html)
+ - [pg\_amproc](reference/catalog/pg_amproc.html)
+ - [pg\_appendonly](reference/catalog/pg_appendonly.html)
+ - [pg\_attrdef](reference/catalog/pg_attrdef.html)
+ - [pg\_attribute](reference/catalog/pg_attribute.html)
+ - [pg\_attribute\_encoding](reference/catalog/pg_attribute_encoding.html)
+ - [pg\_auth\_members](reference/catalog/pg_auth_members.html)
+ - [pg\_authid](reference/catalog/pg_authid.html)
+ - [pg\_cast](reference/catalog/pg_cast.html)
+ - [pg\_class](reference/catalog/pg_class.html)
+ - [pg\_compression](reference/catalog/pg_compression.html)
+ - [pg\_constraint](reference/catalog/pg_constraint.html)
+ - [pg\_conversion](reference/catalog/pg_conversion.html)
+ - [pg\_database](reference/catalog/pg_database.html)
+ - [pg\_depend](reference/catalog/pg_depend.html)
+ - [pg\_description](reference/catalog/pg_description.html)
+ - [pg\_exttable](reference/catalog/pg_exttable.html)
+ - [pg\_filespace](reference/catalog/pg_filespace.html)
+ - [pg\_filespace\_entry](reference/catalog/pg_filespace_entry.html)
+ - [pg\_index](reference/catalog/pg_index.html)
+ - [pg\_inherits](reference/catalog/pg_inherits.html)
+ - [pg\_language](reference/catalog/pg_language.html)
+ - [pg\_largeobject](reference/catalog/pg_largeobject.html)
+ - [pg\_listener](reference/catalog/pg_listener.html)
+ - [pg\_locks](reference/catalog/pg_locks.html)
+ - [pg\_namespace](reference/catalog/pg_namespace.html)
+ - [pg\_opclass](reference/catalog/pg_opclass.html)
+ - [pg\_operator](reference/catalog/pg_operator.html)
+ - [pg\_partition](reference/catalog/pg_partition.html)
+ - [pg\_partition\_columns](reference/catalog/pg_partition_columns.html)
+ - [pg\_partition\_encoding](reference/catalog/pg_partition_encoding.html)
+ - [pg\_partition\_rule](reference/catalog/pg_partition_rule.html)
+ - [pg\_partition\_templates](reference/catalog/pg_partition_templates.html)
+ - [pg\_partitions](reference/catalog/pg_partitions.html)
+ - [pg\_pltemplate](reference/catalog/pg_pltemplate.html)
+ - [pg\_proc](reference/catalog/pg_proc.html)
+ - [pg\_resqueue](reference/catalog/pg_resqueue.html)
+ - [pg\_resqueue\_status](reference/catalog/pg_resqueue_status.html)
+ - [pg\_rewrite](reference/catalog/pg_rewrite.html)
+ - [pg\_roles](reference/catalog/pg_roles.html)
+ - [pg\_shdepend](reference/catalog/pg_shdepend.html)
+ - [pg\_shdescription](reference/catalog/pg_shdescription.html)
+ - [pg\_stat\_activity](reference/catalog/pg_stat_activity.html)
+ - [pg\_stat\_last\_operation](reference/catalog/pg_stat_last_operation.html)
+ - [pg\_stat\_last\_shoperation](reference/catalog/pg_stat_last_shoperation.html)
+ - [pg\_stat\_operations](reference/catalog/pg_stat_operations.html)
+ - [pg\_stat\_partition\_operations](reference/catalog/pg_stat_partition_operations.html)
+ - [pg\_statistic](reference/catalog/pg_statistic.html)
+ - [pg\_stats](reference/catalog/pg_stats.html)
+ - [pg\_tablespace](reference/catalog/pg_tablespace.html)
+ - [pg\_trigger](reference/catalog/pg_trigger.html)
+ - [pg\_type](reference/catalog/pg_type.html)
+ - [pg\_type\_encoding](reference/catalog/pg_type_encoding.html)
+ - [pg\_window](reference/catalog/pg_window.html)
+ - [The hawq\_toolkit Administrative Schema](reference/toolkit/hawq_toolkit.html)
+ - [Checking for Tables that Need Routine Maintenance](reference/toolkit/hawq_toolkit.html#topic2)
+ - [Viewing HAWQ Server Log Files](reference/toolkit/hawq_toolkit.html#topic16)
+ - [Checking Database Object Sizes and Disk Space](reference/toolkit/hawq_toolkit.html#topic38)
+ - [HAWQ Management Tools Reference](reference/cli/management_tools.html)
+ - [analyzedb](reference/cli/admin_utilities/analyzedb.html)
+ - [createdb](reference/cli/client_utilities/createdb.html)
+ - [createuser](reference/cli/client_utilities/createuser.html)
+ - [dropdb](reference/cli/client_utilities/dropdb.html)
+ - [dropuser](reference/cli/client_utilities/dropuser.html)
+ - [gpfdist](reference/cli/admin_utilities/gpfdist.html)
+ - [gplogfilter](reference/cli/admin_utilities/gplogfilter.html)
+ - [gppkg](reference/cli/admin_utilities/gppkg.html)
+ - [hawq activate](reference/cli/admin_utilities/hawqactivate.html)
+ - [hawq check](reference/cli/admin_utilities/hawqcheck.html)
+ - [hawq checkperf](reference/cli/admin_utilities/hawqcheckperf.html)
+ - [hawq config](reference/cli/admin_utilities/hawqconfig.html)
+ - [hawq extract](reference/cli/admin_utilities/hawqextract.html)
+ - [hawq filespace](reference/cli/admin_utilities/hawqfilespace.html)
+ - [hawq init](reference/cli/admin_utilities/hawqinit.html)
+ - [hawq load](reference/cli/admin_utilities/hawqload.html)
+ - [hawq restart](reference/cli/admin_utilities/hawqrestart.html)
+ - [hawq scp](reference/cli/admin_utilities/hawqscp.html)
+ - [hawq ssh](reference/cli/admin_utilities/hawqssh.html)
+ - [hawq ssh-exkeys](reference/cli/admin_utilities/hawqssh-exkeys.html)
+ - [hawq start](reference/cli/admin_utilities/hawqstart.html)
+ - [hawq state](reference/cli/admin_utilities/hawqstate.html)
+ - [hawq stop](reference/cli/admin_utilities/hawqstop.html)
+ - [pg\_dump](reference/cli/client_utilities/pg_dump.html)
+ - [pg\_dumpall](reference/cli/client_utilities/pg_dumpall.html)
+ - [pg\_restore](reference/cli/client_utilities/pg_restore.html)
+ - [psql](reference/cli/client_utilities/psql.html)
+ - [vacuumdb](reference/cli/client_utilities/vacuumdb.html)
+
diff --git a/install/aws-config.md b/install/aws-config.md
new file mode 100644
index 0000000..f5697b1
--- /dev/null
+++ b/install/aws-config.md
@@ -0,0 +1,152 @@
+---
+title: Amazon EC2 Configuration
+---
+
+Amazon Elastic Compute Cloud (EC2) is a service provided by Amazon Web Services (AWS). You can install and configure HAWQ on virtual servers provided by Amazon EC2. The following overview information describes some considerations when installing HAWQ in an Amazon EC2 environment.
+
+## About Amazon EC2 <a id="topic_wqv_yfx_y5"></a>
+
+Amazon EC2 can be used to launch as many virtual servers as you need, configure security and networking, and manage storage. An EC2 *instance* is a virtual server in the AWS cloud virtual computing environment.
+
+EC2 instances are managed by AWS. AWS isolates your EC2 instances from other users in a virtual private cloud (VPC) and lets you control access to the instances. You can configure instance features such as operating system, network connectivity (network ports and protocols, IP addresses), access to the Internet, and size and type of disk storage.
+
+When you launch an instance, you use a preconfigured template known as an Amazon Machine Image (AMI). The AMI packages the bits you need for your server, including the operating system and additional software components. Amazon provides several AMIs from which to choose, or you can create and customize your own.
+
+You launch instances in an Availability Zone of an AWS region. An *Availability Zone* is a distinct location within a region that is engineered to be insulated from failures in other Availability Zones.
+
+For information about Amazon EC2, see the [EC2 User Guide](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html).
+
+## Launching Instances<a id="topic_nhk_df4_2v"></a>
+
+Use the *Amazon EC2 Console* to launch instances and configure, start, stop, and terminate (delete) virtual servers. When you launch an instance, you select, configure, and view the following features in the EC2 Console.
+
+| Feature | Description |
+|--------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Amazon Machine Image (AMI) | A template that contains a software configuration including the operating system, application server, and applications that best suit your purpose. |
+| Instance Type | Varying combinations of CPU, memory, default storage, and networking capacity. |
+| Instance Details | Information about starting, running, and stopping EC2 instances, including number of instances of the same AMI, network information, and EC2 VPC and subnet membership.|
+| Storage Configuration | Tools to adjust and add storage. For example, you can change the size of the root volume or add volumes.|
+| Tagging | An optional (set of) case-sensitive key-value pairs used for managing or organizing a large number of EC2 resources.|
+| Security Groups | A set of firewall rules that control the network traffic for instances. For external access to an instance with `ssh`, you will create a rule that enables `ssh` for inbound network traffic. |
+
+
+### Select AMI Type<a id="topic_selamitype"></a>
+When Configuring a HAWQ virtual instance, select an AMI that runs an operating system supported by HAWQ. Refer to the Release Notes for the HAWQ version that you are installing for a list of supported OS platforms.
+
+**Note**: To create and launch a customized AMI, see [About Amazon Machine Images](#topic_n3y_4gx_y5)
+
+
+### Select and Configure EC2 Instance Type<a id="topic_selcfginsttype"></a>
+When using Amazon Elastic Block Store (EBS), HAWQ supports only EBS-Optimized instance types. See [Configure Storage](#hawq_storage) for information about HAWQ storage requirements. For information about EBS-Optimized instances, see the Amazon [EBS](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSOptimized.html) documentation.
+
+For sufficient network performance, the instance type must also support EC2 enhanced networking. For information about this feature, see [Amazon Enhanced Networking on Linux Instances](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking.html).
+
+The HAWQ cluster instances should be in a single VPC and subnet. Instances are always assigned a VPC internal IP address and can be assigned a public IP address for external and Internet access.
+
+The internal IP address is used for HAWQ communication between hosts. You can also use the internal IP address to access an instance from another instance within the EC2 VPC. For information about configuring launched instances to communicate with each other, see [Working with EC2 Instances](#topic_mj4_524_2v).
+
+Both a public IP address for the instance and an Internet gateway configured for the EC2 VPC are required to access the instance from an external source and for the instance to access the Internet. (Internet access is required when installing Linux packages.)
+
+When you launch a set of instances, you can enable or disable the automatic assignment of public IP addresses when the instances are started. If automatic assignment of public IP addresses is enabled, instances are always assigned a public IP address when started. If automatic assignment of public IP addresses is disabled, you can temporarily associate an [EC2 Elastic IP Address](#task_yd3_sd4_2v) with the instance.
+
+To control whether a public IP address is assigned when launching an instance, see the Amazon documentation on [Subnet Public IP Addressing](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-ip-addressing.html#subnet-public-ip).
+
+
+###Configure Storage<a id="hawq_storage"></a>
+**HAWQ** supports either EC2 instance store or Amazon EBS in production environments.
+
+ - EC2 *instance store* provides temporary block-level storage. This storage is located on disks that are physically attached to the host computer. Powering off the instance causes data loss when using instance store storage. Soft reboots preserve instance store data. EC2 instance store can provide higher and more consistent I/O performance.
+
+ For additional information about EC2 instance store, see [Amazon EC2 Instance Store](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html).
+ - *EBS* provides block level storage volumes with long-term persistence. HAWQ supported configurations using EBS must be a RAID of Amazon EBS volumes mounted with the XFS file system. All other file systems are explicitly not supported.
+There are several classes of EBS. For HAWQ, select an EBS volume type of `gp2` or `io1`. See the Amazon [Block Device Mapping](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/block-device-mapping-concepts.html) documentation for more information.
+
+For additional information about HAWQ requirements on the Amazon storage types, see the [Notes](#topic_kwk_ygx_y5).
+
+
+
+##Working with EC2 Instances <a id="topic_mj4_524_2v"></a>
+
+After launching an EC2 instance, connect to and configure the instance. The *Instances* page of the EC2 Console lists the running instances and their associated network access information. If the instance does not have a public IP address, you can create an Elastic IP address and associate it with the instance. See [About Elastic IP Addresses](#task_yd3_sd4_2v) below.
+
+To access EC2 instances, AWS uses public-key cryptography to secure the login information for your instance. A Linux instance has no password; you use a key pair to log in to your instance securely. You identify the name of the key pair when you launch your instance, and then provide the private key when you log in using SSH. See the Amazon [EC2 Key Pairs](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html) documentation for more detailed information.
+
+A key pair for an EC2 instance consists of a *public key* that AWS stores, and a *private key file* that you maintain. Together, they allow you to connect to your instance securely. The private key file name typically has a `.pem` suffix.
+
+This example logs into an into EC2 instance from an external location with the private key file `my-test.pem` as user `ec2-user`. (`ec2-user` is the default user for some Linux AMI templates.) In this example, the instance is configured with the public IP address `52.36.113.82` and the private key file resides in the current directory.
+
+```shell
+$ ssh -i my-test.pem ec2-user@52.36.113.82
+```
+
+Before installing HAWQ, set up the EC2 instances as you would local host server machines. Configure the host operating system, configure host network information (for example, update the `/etc/hosts` file), set operating system parameters, and install operating system packages. For information about how to prepare your operating system environment for HAWQ, see [Select HAWQ Host Machines](../install/select-hosts.html).
+
+These example commands log in to an EC2 instance and use `yum` to install the `ed`, `unzip`, and `vim` Linux packages on the instance.
+
+```shell
+$ ssh -i my-test.pem ec2-user@52.36.113.82
+ec2inst$ sudo yum install -y ed unzip vim
+```
+
+##About Amazon Machine Images (AMIs) <a id="topic_n3y_4gx_y5"></a>
+
+An Amazon Machine Image (AMI) is a template for a specific software configuration (for example, the collection of an operating system, application servers, and applications). From an AMI, you launch an *instance*, which is a copy of the AMI running as a virtual server in the cloud. You can launch multiple instances of an AMI.
+
+After you launch an instance, it acts like a traditional host; you can interact with it as you would any computer. You have complete control of your instances. For example, you can use `sudo` to run commands that require root privileges.
+
+You can create a new Amazon EBS-backed Linux AMI from an instance that you've already launched and customized to your needs. First, create and register a new AMI based on the customized AMI. You can then use the new AMI to launch instances with these customizations.
+
+For information about AMIs, see the Amazon [AMI User Guide](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html) documentation.
+
+
+## About Amazon Elastic IP Addresses<a id="task_yd3_sd4_2v"></a>
+
+An EC2 Elastic IP address is a public IP address that you can create for your account. You can associate it to and disassociate it from instances as you require; it's allocated to your account until you choose to release it.
+
+Your default VPC is configured with an Internet gateway. When you allocate an EC2 Elastic IP address, AWS configures the VPC to allow internet access to the IP address using the gateway.
+
+To enable an instance in your VPC to communicate with the Internet, it must have a public IP address or an EC2 Elastic IP address that's associated with a private IP address on your instance.
+
+To ensure that your instances can communicate with the Internet, you must also attach an Internet gateway to your EC2 VPC. For information about VPC Internet Gateways, see the Amazon [Internet Gateways] (http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Internet_Gateway.html) documentation.
+
+For more detailed information about EC2 Elastic IP addresses and how to use them, see the [Amazon Elastic IP Address](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html).
+
+
+##Notes<a id="topic_kwk_ygx_y5"></a>
+
+- Supported HAWQ storage configurations with Amazon EBS include a RAID of Amazon EBS volumes mounted with the XFS file system.
+
+- Virtual devices for instance store volumes for HAWQ EC2 instance store instances are named `ephemeralN` (where *N* is between 0 and 23). CentOS instance store block device names appear as `/dev/xvdletter` (where *letter* is a lower case letter of the alphabet).
+
+ Two EC2 instance types configured with instance store showed acceptable performance. These include the `d2.8xlarge` instance type configured with four `raid0` volumes of 6 disks each, and the `i2.8xlarge` instance type configured with two `raid0` volumes of 4 disks.
+
+- To allow access to/from a source external to a VPC, include and open the following ports in the appropriate security group.
+
+ | Port | Application |
+ |-------|-------------------------------------|
+ | 22 | ssh - secure connect to other hosts |
+ | 8080 | Ambari - HAWQ admin/config web console |
+
+- You can configure a non-default VPC with an internet gateway and allocate an Elastic IP address for the VPC. AWS will automatically configure the Elastic IP for internet access.
+
+- A *placement group* is a logical grouping of instances within a single Availability Zone. Using placement groups enables applications to participate in a low-latency, 10 Gbps network. Placement groups are recommended for applications that benefit from low network latency, high network throughput, or both.
+
+ Placement Groups could be used to separate EC2 instances from other instances. While configuring instances in different placement groups can improve performance, it may create a configuration where an instance in a placement group cannot be replaced.
+
+ See [Amazon Placement Group](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html) for additional information.
+
+- Amazon EC2 provides enhanced networking capabilities on some instance types by using single root I/O virtualization (SR-IOV). Enabling enhanced networking on your instance results in higher performance (packets per second), lower latency, and lower jitter.
+
+ To enable enhanced networking on your Red Hat and CentOS RHEL/CentOS instance, you must ensure that the kernel has the `ixgbevf` module version 2.14.2 or higher installed and that the `sriovNetSupport` attribute is set.
+
+##Additional References<a id="topic_hgz_zwy_bv"></a>
+
+Links to related AWS and EC2 features and information.
+
+- [Amazon Web Services](https://aws.amazon.com)
+- [Connecting to an EC2 instance](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstancesLinux.html)
+- [Amazon VPC](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Introduction.html)
+- [Amazon EC2 best practices](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-best-practices.html)
+- [Amazon EC2 and VPC command line interface](http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/Welcome.html)
+- [Amazon S3 \(secure, durable, scalable object storage\)](https://aws.amazon.com/s3)
+- [AWS CloudFormation](https://aws.amazon.com/cloudformation) - With AWS CloudFormation, you can create templates to simplify provisioning and management of related AWS resources.
\ No newline at end of file
diff --git a/install/index.md.erb b/install/index.md.erb
deleted file mode 100644
index c9c060b..0000000
--- a/install/index.md.erb
+++ /dev/null
@@ -1,11 +0,0 @@
----
-title: Installing Pivotal HDB
----
-
-This section describes how to install Pivotal HDB 2.0 using either the Ambari graphical management tool, or using command-line utilities. It contains the procedures:
-
-* <a href="./select-hosts.html" class="subnav">Selecting HAWQ Host Machines</a>
-* <a href="./install-ambari.html" class="subnav">Installing HAWQ Using Ambari</a>
-* <a href="./install-cli.html" class="subnav">Install HAWQ from the Command-Line (Optional)</a>.
-
-**Note:** Use the same installation method (Ambari or CLI) that you used to install the HDP cluster.
diff --git a/install/install-ambari.md b/install/install-ambari.md
deleted file mode 100644
index bfc0ed6..0000000
--- a/install/install-ambari.md
+++ /dev/null
@@ -1,277 +0,0 @@
----
-title: Install Apache HAWQ using Ambari
----
-
-## Prerequisites <a id="section_mqs_f3j_5r"></a>
-
-- Install a compatible version of HDP and Ambari, and ensure that your HDP system is fully functional. See the [Release Notes](/hdb20/releasenotes/HAWQ20ReleaseNotes.html#topic_dhh_2jx_yt) for more information about HDFS compatibility.
-- Select and prepare all host machines that will run the HAWQ and PXF services. See [Select HAWQ Host Machines](select-hosts.html).
-
-## Procedure <a id="section_kwy_f3j_5r"></a>
-
-1. Login to the Ambari server host machine as the `root` user.
-2. Create a staging directory where you will download and extract the tarballs for HAWQ and the HAWQ Ambari plug-in. The staging directory and all the directories above it must be readable and executable by the system user that runs the httpd process \(typically `apache`\). Make the directory readable and executable by all users. For example:
-
- ```
- $ mkdir /staging
- $ chmod a+rx /staging
- ```
-
- **Note:** Do not use the /tmp directory as a staging directory because files under /tmp can be removed at any time.
-
-3. Download the required Pivotal software tarball files from [Pivotal Network](https://network.pivotal.io/products/pivotal-hdb), saving them into the staging directory that you just created. The required tarball files are:
-
- |**Stack Name**|**Filename**|**Description**|
- |--------------|------------|---------------|
- |HDB-2.0.0.0|hdb-2.0.0.0-<build>.tar.gz|Pivotal HDB is a parallel SQL query engine that includes features from Apache HAWQ \(Incubating\) such as PXF.|
- |HDB-AMBARI-PLUGIN-2.0.0|hdb-ambari-plugin-2.0.0-hdp-<build>.tar.gz|The HAWQ plug-in provides Ambari installation and monitoring functionality for Apache HAWQ \(Incubating\).|
-
-4. Extract each tarball file into the staging directory:
-
- ```
- $ tar -xvzf /staging/hdb-2.0.0.0-*.tar.gz -C /staging/
- $ tar -xvzf /staging/hdb-ambari-plugin-2.0.0-hdp-*.tar.gz -C /staging/
- ```
-
-4. Install and/or run the HTTP service if it is not already running:
-
- ```
- $ yum install httpd
- $ service httpd start
- ```
-
-5. Each tarball is an archived yum repository and has a setup\_repo.sh script. The script creates a symlink from the document root of the httpd server \(/var/www/html\) to the directory where the tarball was extracted. On the host that will be used as a YUM repo, execute the setup\_repo.sh script that is shipped as a part of each tarball file:
-
- ```
- $ cd /staging/hdb*
- $ ./setup_repo.sh
- $ cd /staging/hdb-ambari-plugin*
- $ ./setup_repo.sh
- ```
-
-6. Update the Yum repo to install the HAWQ Ambari plug-in:
-
- ```
- $ yum install hdb-ambari-plugin
- ```
-
-6. Restart the Ambari server:
-
- ```
- $ ambari-server restart
- ```
-
- **Note:** You must restart the Ambari server before proceeding. The above command restarts only the Ambari server, but leaves other Hadoop services running.
-
-7. If you have already installed a HDP cluster and are adding HDB to the existing cluster, execute the following script to add the HDB repository to the Ambari server. (This step is not required if you are installing a new HDP cluster and HDB together at the same time.):
-
- ```
- $ cd /var/lib/hawq
- $ ./add_hdb_repo.py -u admin -p admin
- ```
-
- **Note:** Substitute the correct Ambari administrator password for your system.
-
-8. Access the Ambari web console at http://ambari.server.hostname:8080, and login as the "admin" user. \(The default password is also "admin".\) Verify that the HDB component is now available.
-11. Select **HDFS**, then click the **Configs** tab.
-12. Customize the HDFS configuration by following these steps:
- 1. On the **Settings** tab, change the DataNode setting **DataNode max data transfer threads** \(dfs.datanode.max.transfer.threads parameter \) to *40960*.
- 1. Select the **Advanced** tab and expand **DataNode**. Ensure that the **DataNode directories permission** \(dfs.datanode.data.dir.perm parameter\) is set to *750*.
- 1. Expand the **General** tab and change the **Access time precision** \(dfs.namenode.accesstime.precision parameter\) to *0*.
- 1. Expand **Advanced hdfs-site**. Set the following properties to their indicated values.
-
- **Note:** If a property described below does not appear in the Ambari UI, select **Custom hdfs-site** and click **Add property...** to add the property definition and set it to the indicated value.
-
- |Property|Setting|
- |--------|-------|
- |**dfs.allow.truncate**|true|
- |**dfs.block.access.token.enable**|*false* for an unsecured HDFS cluster, or *true* for a secure cluster|
- |**dfs.block.local-path-access.user**|gpadmin|
- |**HDFS Short-circuit read** \(**dfs.client.read.shortcircuit**\)|true|
- |**dfs.client.socket-timeout**|300000000|
- |**dfs.client.use.legacy.blockreader.local**|false|
- |**dfs.datanode.handler.count**|60|
- |**dfs.datanode.socket.write.timeout**|7200000|
- |**dfs.namenode.handler.count**|600|
- |**dfs.support.append**|true|
-
- **Note:** HAWQ requires that you enable `dfs.allow.truncate`. The HAWQ service will fail to start if `dfs.allow.truncate` is not set to "true."
-
- 2. Expand **Advanced core-site**, then set the following properties to their indicated values:
-
- **Note:** If a property described below does not appear in the Ambari UI, select **Custom core-site** and click **Add property...** to add the property definition and set it to the indicated value.
-
- |Property|Setting|
- |--------|-------|
- |**ipc.client.connection.maxidletime**|3600000|
- |**ipc.client.connect.timeout**|300000|
- |**ipc.server.listen.queue.size**|3300|
-
-13. Click **Save** and enter a name for the configuration change (for example, *HAWQ prerequisites*). Click **Save** again, then **OK**.
-13. If Ambari indicates that a service must be restarted, click **Restart** and allow the service to restart before you continue.
-13. Select **Actions \> Add Service** on the home page.
-14. Select both **HAWQ** and **PXF** from the list of services, then click **Next** to display the Assign Masters page.
-15. Select the hosts that should run the HAWQ Master and HAWQ Standby Master, or accept the defaults. The HAWQ Master and HAWQ Standby Master must reside on separate hosts. Click **Next** to display the Assign Slaves and Clients page.
->
-
- **Note:** Only the **HAWQ Master** and **HAWQ Standby Master** entries are configurable on this page. NameNode, SNameNode, ZooKeeper and others may be displayed for reference, but they are not configurable when adding the HAWQ and PXF services.
->
-
- **Note:** The HAWQ Master component must not reside on the same host that is used for Hive Metastore if the Hive Metastore uses the new PostgreSQL database. This is because both services attempt to use port 5432. If it is required to co-locate these components on the same host, provision a PostgreSQL database beforehand on a port other than 5432 and choose the “Existing PostgreSQL Database” option for the Hive Metastore configuration. The same restriction applies to the admin host, because neither the HAWQ Master nor the Hive Metastore can run on the admin host where the Ambari Server is installed.
-
-16. On the Assign Slaves and Clients page, choose the hosts that will run HAWQ segments and PXF, or accept the defaults. The Add Service Wizard automatically selects hosts for the HAWQ and PXF services based on available Hadoop services.
->
-
- **Note:** PXF must be installed on the HDFS NameNode, the Standby NameNode \(if configured\), *and* on each HDFS DataNode. A HAWQ segment must be installed on each HDFS DataNode.
->
-
- Click **Next** to continue.
-
-17. On the Customize Services page, the **Settings** tab configures basic properties of the HAWQ cluster. In most cases you can accept the default values provided on this page. Several configuration options may require attention depending on your deployment:
- * **HAWQ Master Directory**, **HAWQ Segment Directory**: This specifies the base path for the HAWQ master or segment data directory.
- * **HAWQ Master Temp Directories**, **HAWQ Segment Temp Directories**: HAWQ temporary directories are used for spill files. Enter one or more directories in which the HAWQ Master or a HAWQ segment stores these temporary files. Separate multiple directories with a comma. Any directories that you specify must already be available on all host machines. If you do not specify master or segment temporary directories, temporary files are stored in `/tmp/hawq/[master|segment]`.<br/><br/>As a best practice, use multiple master and segment temporary directories on separate, large disks (2TB or greater) to load balance writes to temporary files \(for example, `/disk1/tmp,/disk2/tmp`\). For a given query, HAWQ will use a separate temp directory (if available) for each virtual segment to store spill files. Multiple HAWQ sessions will also use separate temp directories where available to avoid disk contention. If you configure too few temp directories, or you place multiple temp directories on the same disk, you increase the risk of disk contention or running out of disk space when multiple virtual segments target the same disk. Each HAWQ segment node can have 6 virtual segments.
- * **Resource Manager**: Select the resource manager to use for allocating resources in your HAWQ cluster. If you choose **Standalone**, HAWQ exclusively uses resources from the whole cluster. If you choose **YARN**, HAWQ contacts the YARN resource manager to negotiate resources. You can change the resource manager type after the initial installation. You will also have to configure some YARN-related properties in step 22. For more information on using YARN to manage resources, see [Managing Resources](/200/hawq/resourcemgmt/HAWQResourceManagement.html).
-
- **Caution:** If you are installing HAWQ in secure mode (Kerberos-enabled), then set **Resource Manager** to **Standalone** to avoid encountering a known installation issue. You can enable YARN mode post-installation if YARN resource management is desired in HAWQ.
- * **VM Overcommit**: Set this value according to the instructions in the [System Requirements](/20/requirements/system-requirements.html) document.
-
-17. Click the **Advanced** tab and enter a **HAWQ System User Password**. Retype the password where indicated.
-
-17. \(Optional.\) On the **Advanced** tab, you can change numerous configuration properties for the HAWQ cluster. Hover your mouse cursor over the entry field to display help for the associated property. Default values are generally acceptable for a new installation. The following properties are sometimes customized during installation:
-
- |Property|Action|
- |--------|------|
- |**General > HAWQ DFS URL**|The URL that HAWQ uses to access HDFS|
- |**General > HAWQ Master Port**|Enter the port to use for the HAWQ master host or accept the default, 5432. **CAUTION:** If you are installing HAWQ in a single-node environment \(or when the Ambari server and HAWQ are installed the same node\) then *you cannot accept the default port*. Enter a unique port for the HAWQ master|
- |**General > HAWQ Segment Port**|The base port to use for HAWQ segment hosts|
-
-1. If you selected YARN as the **Resource Manager**, then you must configure several YARN properties for HAWQ. On the **Advanced** tab of HAWQ configuration, enter values for the following parameters:
-
- |Property|Action|
- |--------|------|
- |**Advanced hawq-site > hawq\_rm\_yarn\_address**|Specify the address and port number of the YARN resource manager server \(the value of `yarn.resourcemanager.address`\). For example: rm1.example.com:8050|
- |**Advanced hawq-site > hawq\_rm\_yarn\_queue\_name**|Specify the YARN queue name to use for registering the HAWQ resource manager. For example: `default` **Note:** If you specify a custom queue name other than `default`, you must configure the YARN scheduler and custom queue capacity, either through Ambari or directly in YARN's configuration files. See [Integrating YARN with HAWQ](/20/resourcemgmt/YARNIntegration.html) for more details. |
- |**Advanced hawq-site > hawq\_rm\_yarn\_scheduler\_address**|Specify the address and port number of the YARN scheduler server \(the value of `yarn.resourcemanager.scheduler.address`\). For example: rm1.example.com:8030|
-
- If you have enabled high availability for YARN, then verify that the following values have been populated correctly in HAWQ:
-
- |Property|Action|
- |--------|------|
- |**Custom yarn-client > yarn.resourcemanager.ha**|Comma-delimited list of the fully qualified hostnames of the resource managers. When high availability is enabled, YARN ignores the value in hawq\_rm\_yarn\_address and uses this property’s value instead. For example, `rm1.example.com:8050,rm2.example.com:8050` |
- |**Custom yarn-client > yarn.resourcemanager.scheduler.ha**|Comma-delimited list of fully qualified hostnames of the resource manager schedulers. When high availability is enabled, YARN ignores the value in hawq\_rm\_yarn\_scheduler\_address and uses this property’s value instead. For example,`rm1.example.com:8030,rm2.example.com:8030` |
-
-18. Click **Next** to continue the installation. (Depending on your cluster configuration, Ambari may recommend that you change other properties before proceeding.)
-18. Review your configuration choices, then click **Deploy** to begin the installation. Ambari now begins to install, start, and test the HAWQ and PXF configuration. During this procedure, you can click on the **Message** links to view the console output of individual tasks.
-
-18. Click **Next** after all tasks have completed. Review the summary of the install process, then click **Complete**. Ambari may indicate that components on cluster need to be restarted. Choose **Restart > Restart All Affected** if necessary.
-19. To verify that HAWQ is installed, login to the HAWQ master as `gpadmin` and perform some basic commands:
-
- 1. Source the `greenplum_path.sh` file to set your environment for HAWQ:
-
- ```
- $ source /usr/local/hawq/greenplum_path.sh
- ```
-
- 1. If you use a custom HAWQ master port number, set it in your environment. For example:
-
- ```
- $ export PGPORT=5432
- ```
-
- 1. Start the `psql` interactive utility, connecting to the postgres database:
-
- ```
- $ psql -d postgres
- psql (8.2.15)
- Type "help" for help.
- postgres=#
- ```
-
- 3. Create a new database and connect to it:
-
- ```
- postgres=# create database mytest;
- CREATE DATABASE
- postgres=# \c mytest
- You are now connected to database "mytest" as user "*username*".
- ```
-
- 4. Create a new table and insert sample data:
-
- ```
- mytest=# create table t (i int);
- CREATE TABLE
- mytest=# insert into t select generate_series(1,100);
- ```
-
- 5. Activate timing and perform a simple query:
-
- ```
- mytest=# \timing
- Timing is on.
- mytest=# select count(*) from t;
- count
- -------
- 100
- (1 row)
- Time: 7.266 ms
- ```
-
-## Post-Install Procedure for Hive and HBase on HDP<a id="post-install-pxf"></a>
-
-In order to use the installed PXF service with HBase on a HDP cluster, you must manually add the path to the `pxf-hbase.jar` file to the `HBASE_CLASSPATH` environment variable and restart HBase.
-
-If you are using Kerberos to secure Hive and HBase, you must configure proxy users, enable user impersonation, and configure PXF access to tables.
-
-Follow this procedure to make the required changes:
-
-1. Use either a text editor or the Ambari Web interface to edit the `hbase-env.sh` file, and add the line:
-
- ```
- export HBASE_CLASSPATH=${HBASE_CLASSPATH}:/usr/lib/pxf/pxf-hbase.jar
- ```
-
- **Note:** Restart HBase after adding the PXF service in order to load the newly-installed PXF JAR file.
-
-2. (Optional.) For secure Hive installations, use either a text editor or the Ambari Web interface to edit the `hive-site.xml` file, and add the property:
-
- ```
- <property>
- <name>hive.server2.enable.impersonation</name>
- <description>Enable user impersonation for HiveServer2</description>
- <value>true</value>
- </property>
- ```
-
-3. (Optional.) For secure Hive and HBase installations, use either a text editor or the Ambari Web interface to edit the core-site.xml file, and add the properties:
-
- ```
- <property>
- <name>hadoop.proxyuser.hive.hosts</name>
- <value>*</value>
- </property>
-
- <property>
- <name>hadoop.proxyuser.hive.groups</name>
- <value>*</value>
- </property>
-
- <property>
- <name>hadoop.proxyuser.hbase.hosts</name>
- <value>*</value>
- </property>
-
- <property>
- <name>hadoop.proxyuser.hbase.groups</name>
- <value>*</value>
- </property>
- ```
-
-4. Restart both Hive and HBase to use the updated classpath and new properties.
-5. In order to use PXF with HBase or Hive tables, you must grant the `pxf` user read permission on those tables:
- * For HBase, use the `GRANT` command for each table that you want to access with PXF. For example:
-
- ```
- hbase(main):001:0> grant 'pxf', 'R', 'my_table'
- ```
- * Because Hive uses the HDFS ACLs for access control, ensure that the pxf has read permission on all of the HDFS directories that map to your database, tables, and partitions.
diff --git a/install/install-cli.md b/install/install-cli.md
deleted file mode 100644
index fe42b49..0000000
--- a/install/install-cli.md
+++ /dev/null
@@ -1,683 +0,0 @@
----
-title: Install HAWQ from the Command Line (Optional)
----
-
-This section provides instructions for installing a HAWQ system.
-
-**Note:** Install HAWQ from the command line only if you do not use Ambari to install and manage HDFS. If you use Ambari for HDFS and management, follow the instructions in [Install HAWQ using Ambari](install-ambari.html) instead.
-
-## Prerequisites <a id="cliprereq"></a>
-
-Install a compatible HDFS cluster before you attempt to install HAWQ.
-
-## Prepare Host Machines <a id="topic_eqn_fc4_15"></a>
-
-Configure operating system parameters on each host machine before you begin to install HAWQ.
-
-1. Use a text editor to edit the /etc/sysctl.conf file. Add or edit each of the following parameter definitions to set the required value:
-
- ```
- kernel.shmmax = 1000000000
- kernel.shmmni = 4096
- kernel.shmall = 4000000000
- kernel.sem = 250 512000 100 2048
- kernel.sysrq = 1
- kernel.core_uses_pid = 1
- kernel.msgmnb = 65536
- kernel.msgmax = 65536
- kernel.msgmni = 2048
- net.ipv4.tcp_syncookies = 0
- net.ipv4.ip_forward = 0
- net.ipv4.conf.default.accept_source_route = 0
- net.ipv4.tcp_tw_recycle = 1
- net.ipv4.tcp_max_syn_backlog = 200000
- net.ipv4.conf.all.arp_filter = 1
- net.ipv4.ip_local_port_range = 1281 65535
- net.core.netdev_max_backlog = 200000
- vm.overcommit_memory = 2
- fs.nr_open = 3000000
- kernel.threads-max = 798720
- kernel.pid_max = 798720
- # increase network
- net.core.rmem_max=2097152
- net.core.wmem_max=2097152
- ```
-
- Save the file after making the required changes.
-
-2. Execute the following command to apply your updated /etc/sysctl.conf file to the operating system configuration:
-
- ```
- $ sysctl -p
- ```
-
-3. Use a text editor to edit the /etc/security/limits.conf file. Add the following definitions in the exact order that they are listed:
-
- ```
- * soft nofile 2900000
- * hard nofile 2900000
- * soft nproc 131072
- * hard nproc 131072
- ```
-
- Save the file after making the required changes.
-
-4. Use a text editor to edit the `hdfs-site.xml` file. Ensure that the following HDFS parameters are set to the correct value:
-
- |Property|Setting|
- |--------|-------|
- |**dfs.allow.truncate**|true|
- |**dfs.block.access.token.enable**|*false* for an unsecured HDFS cluster, or *true* for a secure cluster|
- |**dfs.block.local-path-access.user**|gpadmin|
- |**dfs.client.read.shortcircuit**|true|
- |**dfs.client.socket-timeout**|300000000|
- |**dfs.client.use.legacy.blockreader.local**|false|
- |**dfs.datanode.data.dir.perm**|750|
- |**dfs.datanode.handler.count**|60|
- |**dfs.datanode.max.transfer.threads**|40960|
- |**dfs.datanode.socket.write.timeout**|7200000|
- |**dfs.namenode.accesstime.precision**|0|
- |**dfs.namenode.handler.count**|600|
- |**dfs.support.append**|true|
-
-4. Use a text editor to edit the `core-site.xml` file. Ensure that the following HDFS parameters are set to the correct value:
-
- |Property|Setting|
- |--------|-------|
- |**ipc.client.connection.maxidletime**|3600000|
- |**ipc.client.connect.timeout**|300000|
- |**ipc.server.listen.queue.size**|3300|
-
-5. Restart HDFS to apply your configuration changes.
-
-6. Ensure that the `/etc/hosts` file on each cluster node contains the hostname of every other member of the cluster. Consider creating a single, master `/etc/hosts` file and either copying it or referencing it on every host that will take part in the cluster.
-
-## Install the HAWQ Cluster on Multiple Machines <a id="topic_ld1_bh4_15"></a>
-
-Follow this procedure to install the HAWQ cluster on multiple host machines or VMs.
-
-**Note:** If you want to install a cluster on a single host machine or VM, follow the instructions in [Install the HAWQ Cluster on a Single Machine](#topic_ld1_bh4_15) instead.
-
-1. Login to the target master host machine as the root user. If you are logged in as a different user, switch to the root account:
-
- ```
- $ su - root
- ```
-
-2. Download the Apache HAWQ tarball distribution \(hdb-2.0.0.0-18407.tar.gz\) to the local machine.
-3. Unzip the tarball file:
-
- ```
- $ tar xzf hdb-2.0.0.0-18407.tar.gz
- ```
-
- The files are uncompressed into a hdb-2.0.0.0 subdirectory.
-
-4. Execute the RPM installer:
-
- ```
- $ cd hdb-2.0.0.0
- $ rpm -ivh hawq-2.0.0.0-18407.x86_64.rpm
- ```
-
-5. Source the `greenplum_path.sh` file to set your environment for HAWQ. For RPM installations, enter:
-
- ```
- $ source /usr/local/hawq/greenplum_path.sh
- ```
-
- If you downloaded the tarball, substitute the path to the extracted `greenplum_path.sh` file \(for example `/opt/hawq-2.0.0.0/greenplum_path.sh`\).
-
-6. Use a text editor to create a file \(for example, hostfile\) that lists the hostname of the master node, the standby master node, and each segment node used in the cluster. Specify one hostname per line, as in the following example that defines a cluster with a master node, standby master node, and three segments:
-
- ```
- mdw
- smdw
- sdw1
- sdw2
- sdw3
- ```
-
- Create a second text file \(for example, seg\_hosts\) that specifies only the HAWQ segment hosts in your cluster. For example:
-
- ```
- sdw1
- sdw2
- sdw3
- ```
-
- You will use the hostfile and seg\_hosts files in subsequent steps to perform common configuration tasks on multiple nodes of the cluster.
-
-7. HAWQ requires passwordless ssh access to all cluster nodes. Execute the following `hawq` command to set up passwordless ssh on the nodes defined in your hostfile:
-
- ```
- $ hawq ssh-exkeys -f hostfile
- ```
-
-8. Execute the following `hawq` to copy the installation RPM to all nodes defined in your hostfile:
-
- ```
- $ hawq scp -f hostfile hawq-2.0.0.0-18407.x86_64.rpm =:~/
- ```
-
-9. Execute this command to initiate the RPM installation on all cluster hosts:
-
- ```
- $ hawq ssh -f hostfile -e "rpm -ivh ~/hawq-*.rpm"
- ```
-
-10. Create the gpadmin user on all hosts in the cluster:
-
- ```
- $ hawq ssh -f hostfile -e '/usr/sbin/useradd gpadmin'
- $ hawq ssh –f hostfile -e 'echo -e "changeme\changeme" | passwd gpadmin'
- ```
-
-11. Switch to the gpadmin user, and source the greenplum\_path.sh file once again:
-
- ```
- $ su - gpadmin
- $ source /usr/local/hawq/greenplum_path.sh
- ```
-
-12. Execute the following `hawq` command a second time to set up passwordless ssh for the gpadmin user:
-
- ```
- $ hawq ssh-exkeys -f hostfile
- ```
-
-13. Execute this command to confirm that HAWQ was installed on all of the hosts:
-
- ```
- $ hawq ssh -f hostfile -e "ls -l $GPHOME"
- ```
-
-14. Create a directory to use for storing the master catalog data. This directory must have sufficient disk space to store data, and it must be owned by the gpadmin user. For example:
-
- ```
- $ mkdir -p /data/master
- $ chown -R gpadmin /data/master
- ```
-
- If you are using a standby master node, create the same directory on the standby host. For example:
-
- ```
- $ hawq ssh -h smdw -e 'mkdir /data/master'
- $ hawq ssh -h smdw -e 'chown -R gpadmin /data/master'
- ```
-
- Replace `smdw` with the name of the standby host.
-
-15. Use the following `hawq` commands with the seg\_hosts file you created earlier to create the data directory on all of the HAWQ segment hosts:
-
- ```
- $ hawq ssh -f seg_hosts -e 'mkdir /data/segment'
- $ hawq ssh -f seg_hosts -e 'chown gpadmin /data/segment'
- ```
-
-16. HAWQ temporary directories are used for spill files. Create the HAWQ temporary directories on all HAWQ hosts in your cluster. As a best practice, use a directory from each mounted drive available on your machines to load balance writing for temporary files. These drives are typically the same drives that are used by your DataNode service. For example, if you have two drives mounted on `/data1` and `/data2`, you could use `/data1/tmp` and `/data2/tmp` for storing temporary files. If you do not specify master or segment temporary directories, temporary files are stored in `/tmp`.
-
- The following example commands use two disks with the paths /data1/tmp and /data2/tmp:
-
- ```
- $ dirs="/data1/tmp /data2/tmp"
- $ mkdir -p $dirs
- $ chown -R gpadmin $dirs
- $ hawq ssh -h smdw -e "mkdir -p $dirs"
- $ hawq ssh -h smdw -e "chown -R gpadmin $dirs"
- $ hawq ssh -f seg_hosts -e "mkdir -p $dirs"
- $ hawq ssh -f seg_hosts -e "chown -R gpadmin $dirs"
- ```
-
- If you configure too few temp directories, or you place multiple temp directories on the same disk, you increase the risk of disk contention or running out of disk space when multiple virtual segments target the same disk. Each HAWQ segment node can have 6 virtual segments.
-17. Login to the master host as the gpadmin user. Create a customized `$GPHOME/etc/hawq-site.xml` file using the template `$GPHOME/etc/template-hawq-site.xml` file for a multi-node cluster. Your custom hawq-site.xml should include the following modifications:
- 1. Change the `hawq_dfs_url` property definition to use the actual Namenode port number as well as the HAWQ data directory:
-
- ```
- <property>
- <name>hawq_dfs_url</name>
- <value>localhost:8020/hawq_default</value>
- <description>URL for accessing HDFS.</description>
- </property>
-
- ```
-
- If HDFS is configured with NameNode high availability (HA), `hawq_dfs_url` should instead include the service ID that you configured. For example, if you configured HA with the service name `hdpcluster` the entry would be similar to:
-
- ```
- <property>
- <name>hawq_dfs_url</name>
- <value>hdpcluster/hawq_default</value>
- <description>URL for accessing HDFS.</description>
- </property>
-
- ```
-
- Also set gpadmin as the owner of the parent directory HDFS directory you specify. For example:
-
- ```
- $ hdfs dfs -chown gpadmin /
- ```
-
- 2. Configure additional properties to specify the host names, port numbers, and directories used in your system. For example:
-
- |Property|Example Value|
- |--------|-------------|
- |hawq\_master\_address\_host|mdw|
- |hawq\_master\_address\_port|20000|
- |hawq\_standby\_address\_host|smdw|
- |hawq\_segment\_address\_port|40000|
- |hawq\_master\_directory|/data/master|
- |hawq\_segment\_directory|/data/segment|
- |hawq\_master\_temp\_directory|/data1/tmp,/data2/tmp|
- |hawq\_segment\_temp\_directory|/data1/tmp,/data2/tmp|
- |hawq\_global\_rm\_type|none|
-
- **Caution:** If you are installing HAWQ in [secure mode (Kerberos-enabled)](#topic_iwy_5fb_1t), then set `hawq_global_rm_type` to standalone mode (`none`) to avoid encountering a known installation issue. You can enable YARN mode post-installation if YARN resource management is desired in HAWQ.
-
- 3. If you wish to use YARN mode for HAWQ resource management, configure YARN properties for HAWQ. For example, in `$GPHOME/etc/hawq-site.xml`:
-
- |Property|Example Value|Comment|
- |--------|-------------|-------|
- |hawq\_global\_rm\_type|yarn| When set to `yarn`, HAWQ requires that you configure additional YARN configuration parameters (`hawq_rm_yarn_address` or `hawq_rm_yarn_scheduler_address` |
- |hawq\_rm\_yarn\_address|mdw:9980 | This property must match the `yarn.resourcemanager.address` value in `yarn-site.xml`.|
- |hawq\_rm\_yarn\_scheduler\_address|mdw:9981 | This property must match the `yarn.resourcemanager.scheduler.address` value in `yarn-site.xml`.|
-
-
- If you have high availability enabled for YARN resource managers, then you must also configure the HA parameters in `$GPHOME/etc/yarn-client.xml`. For example:
-
-
- |Property|Example Value|Comment|
- |--------|-------------|-------|
- |yarn.resourcemanager.ha| rm1.example.com:8032, rm2.example.com:8032| Comma-delimited list of resource manager hosts. When high availability is enabled, YARN ignores the value in hawq\_rm\_yarn\_address and uses this property's value instead.|
- |yarn.resourcemanager.scheduler.ha|rm1.example.com:8030, rm2.example.com:8030 | Comma-delimited list of scheduler hosts. When high availability is enabled, YARN ignores the value in hawq\_rm\_yarn\_scheduler\_address and uses this property's value instead.|
-
- Replace the example hostnames with the fully qualified domain names of your resource manager host machines.
-
-18. Edit the $GPHOME/etc/slaves file to list all of the segment host names for your cluster. For example:
-
- ```
- sdw1
- sdw2
- sdw3
- ```
-
-8. If your HDFS cluster is configured with NameNode high availability (HA), edit the ` ${GPHOME}/etc/hdfs-client.xml` file on each segment and add the following NameNode properties:
-
- ```
- <property>
- <name>dfs.ha.namenodes.hdpcluster</name>
- <value>nn1,nn2</value>
- </property>
-
- <property>
- <name>dfs.namenode.http-address.hdpcluster.nn1</name>
- <value>ip-address-1.mycompany.com:50070</value>
- </property>
-
- <property>
- <name>dfs.namenode.http-address.hdpcluster.nn2</name>
- <value>ip-address-2.mycompany.com:50070</value>
- </property>
-
- <property>
- <name>dfs.namenode.rpc-address.hdpcluster.nn1</name>
- <value>ip-address-1.mycompany.com:8020</value>
- </property>
-
- <property>
- <name>dfs.namenode.rpc-address.hdpcluster.nn2</name>
- <value>ip-address-2.mycompany.com:8020</value>
- </property>
-
- <property>
- <name>dfs.nameservices</name>
- <value>hdpcluster</value>
- </property>
- ```
-
- In the listing above:
- * Replace `hdpcluster` with the actual service ID that is configured in HDFS.
- * Replace `ip-address-2.mycompany.com:50070` with the actual NameNode RPC host and port number that is configured in HDFS.
- * Replace `ip-address-1.mycompany.com:8020` with the actual NameNode HTTP host and port number that is configured in HDFS.
- * The order of the NameNodes listed in `dfs.ha.namenodes.hdpcluster` is important for performance, especially when running secure HDFS. The first entry (`nn1` in the example above) should correspond to the active NameNode.
-
-19. Synchronize the customized hawq-site.xml and slaves files to all cluster nodes:
-
- ```
- $ hawq scp -f hostfile hawq-site.xml slaves =:$GPHOME/etc/
- ```
-
-21. Finally, initialize and start the new HAWQ cluster using the command:
-
- ```
- $ hawq init cluster
- ```
-
- After the cluster starts, you can follow the instructions in [Validate the Installation](#topic_opr_3gp_15).
-
-
-## Install the HAWQ Cluster on a Single Machine <a id="topic_uqh_wg4_15"></a>
-
-Follow this procedure to install HAWQ software on a single host machine.
-
-1. Login to the target machine as the root user. If you are logged in as a different user, switch to the root account:
-
- ```
- $ su - root
- ```
-
-2. Download the HAWQ tarball distribution \(tar xzf hdb-2.0.0.0-18407.tar.gz\) to the local machine.
-3. Unzip the tarball file:
-
- ```
- $ tar xzf hdb-2.0.0.0-18407.tar.gz
- ```
-
- The files are uncompressed into a `hdb-2.0.0.0` subdirectory.
-
-4. Execute the RPM installer:
-
- ```
- $ cd hdb-2.0.0.0
- $ rpm -ivh hawq-2.0.0.0-18407.x86_64.rpm
- ```
-
-5. Switch to the gpadmin user:
-
- ```
- $ su - gpadmin
- ```
-
-6. Source the greenplum\_path.sh file to set your environment for . For RPM installations, enter:
-
- ```
- $ source /usr/local/hawq/greenplum_path.sh
- ```
-
- If you downloaded the tarball, substitute the path to the extracted greenplum\_path.sh file \(for example /opt/hawq-2.0.0.0/greenplum\_path.sh\).
-
-7. HAWQ requires passwordless ssh access to all cluster nodes, even on a single-node cluster installation. Execute the following `hawq` command to exchange keys and enable passwordless SSH to localhost:
-
- ```
- $ hawq ssh-exkeys -h localhost
- ```
-
-8. If your HDFS Namenode does not use the default port, 8020, then open the $GPHOME/etc/hawq-site.xml file with a text editor and modify the following property definition to use the actual Namenode port number:
-
- ```
- <property>
- <name>hawq_dfs_url</name>
- <value>localhost:8020/hawq_default</value>
- <description>URL for accessing HDFS.</description>
- </property>
-
- ```
-
- Also ensure that the gpadmin user has both read and write access to the parent directory specified with `hawq_dfs_url`. For example:
-
- ```
- $ hdfs dfs -chown gpadmin hdfs://localhost:8020/
- ```
-
-9. Finally, initialize and start the new HAWQ cluster using the command:
-
- ```
- $ hawq init cluster
- ```
-
- After the cluster starts, you can follow the instructions in [Validate the Installation](#topic_opr_3gp_15).
-
-
-## \(Optional\) Enable Kerberos <a id="topic_iwy_5fb_1t"></a>
-
-The following additional steps are necessary only if you manage your system manually \(without using Ambari\), and you enabled Kerberos security for HDFS.
-
-For manual installations, perform these additional steps after you complete the previous procedure:
-
-1. Ensure that the HDFS parameter `dfs.block.access.token.enable` is set to true for the secure HDFS cluster. This property can be set within Ambari via **Services \> HDFS \> Configs \> Advanced hdfs-site \> dfs.block.access.token.enable**. After modifying this parameter, you must restart HDFS.
-2. Login to the Kerberos Key Distribution Center \(KDC\) server as the `root` user.
-3. Use `kadmin.local` to create a new principal for the `postgres` user. This will be used by the HAWQ master segment host:
-
- ```
- $ kadmin.local -q “addprinc -randkey postgres@LOCAL.DOMAIN”
- ```
-
-4. Use `kadmin.local` to generate a Kerberos service principal for all other hosts that will run a HAWQ segment with the PXF service. The service principal should be of the form `name/role@REALM` where:
- - name is the PXF service user name \(`pxf`\). Use the same name for each HAWQ host.
- - role is the DNS resolvable, fully-qualified hostname of the HAWQ host machine \(output of `hostname -f` command\).
- - REALM is the Kerberos Distribution Center \(KDC\) realm used in the Hadoop cluster \(for example, `LOCAL.DOMAIN`\).
-
- You can generate all principals on the KDC server. For example, these commands add service principals for three HAWQ nodes on the hosts host1.example.com, host2.example.com, and host3.example.com:
-
- ```
- $ kadmin.local -q "addprinc -randkey pxf/host1.example.com@LOCAL.DOMAIN"
- $ kadmin.local -q "addprinc -randkey pxf/host2.example.com@LOCAL.DOMAIN"
- $ kadmin.local -q "addprinc -randkey pxf/host3.example.com@LOCAL.DOMAIN"
- ```
- Repeat this step for each host that runs a PXF service. Substitute the fully-qualified hostname of each host machine and the realm name used in your configuration.
->
-
- **Note:** As an alternative, if you have a hosts file that lists the fully-qualified domain name of each cluster host \(one host per line\), then you can generate principals using the command:
-
- ```
- $ for HOST in $\(cat hosts\) ; do sudo kadmin.local -q "addprinc -randkey pxf/$HOST@LOCAL.DOMAIN" ; done
- ```
-
-5. Generate a keytab file for each principal that you created \(for the `postgres` and for each `pxf` service principal\). You can store the keytab files in any convenient location \(this example uses the directory /etc/security/keytabs\). You will deploy the service principal keytab files to their respective HAWQ host machines in a later step:
-
- ```
- $ kadmin.local -q “xst -k /etc/security/keytabs/hawq.service.keytab postgres@LOCAL.DOMAIN”
- $ kadmin.local -q “xst -k /etc/security/keytabs/pxf-host1.service.keytab pxf/host1.example.com@LOCAL.DOMAIN”
- $ kadmin.local -q “xst -k /etc/security/keytabs/pxf-host2.service.keytab pxf/host2.example.com@LOCAL.DOMAIN”
- $ kadmin.local -q “xst -k /etc/security/keytabs/pxf-host3.service.keytab pxf/host3.example.com@LOCAL.DOMAIN”
- $ kadmin.local -q “listprincs”
- ```
-
- Repeat the `xst` command as necessary to generate a keytab for each HAWQ service principal that you created in the previous step.
-
-6. The HAWQ master server also requires a hdfs.headless.keytab file for the HDFS principal to be available under /etc/security/keytabs. If this file does not already exist, generate it using the command:
-
- ```
- $ kadmin.local -q “addprinc -randkey hdfs@LOCAL.DOMAIN”
- $ kadmin.local -q “xst -k /etc/security/keytabs/hdfs.headless.keytab hdfs@LOCAL.DOMAIN”
- ```
-
- **Note:** HAWQ initialization will fail if the above keytab file is not available.
-
-7. Copy the HAWQ service keytab file \(and the hdfs.headless.keytab file if you created one\) to the HAWQ master segment host:
-
- ```
- $ scp /etc/security/keytabs/hawq.service.keytab hawq_master_fqdn:/etc/security/keytabs/hawq.service.keytab
- $ scp /etc/security/keytabs/hdfs.headless.keytab hawq_master_fqdn:/etc/security/keytabs/hdfs.headless.keytab
- ```
-
-8. Change the ownership and permissions on hawq.service.keytab \(and on hdfs.headless.keytab if you copied it\) as follows:
-
- ```
- $ ssh hawq_master_fqdn chown gpadmin:gpadmin /etc/security/keytabs/hawq.service.keytab
- $ ssh hawq_master_fqdn chmod 400 /etc/security/keytabs/hawq.service.keytab
- $ ssh hawq_master_fqdn chown hdfs:hdfs /etc/security/keytabs/hdfs.headless.keytab
- $ ssh hawq_master_fqdn chmod 440 /etc/security/keytabs/hdfs.headless.keytab
- ```
-
-9. Copy the keytab file for each service principal to its respective HAWQ host:
-
- ```
- $ scp /etc/security/keytabs/pxf-host1.service.keytab host1.example.com:/etc/security/keytabs/pxf.service.keytab
- $ scp /etc/security/keytabs/pxf-host2.service.keytab host2.example.com:/etc/security/keytabs/pxf.service.keytab
- $ scp /etc/security/keytabs/pxf-host3.service.keytab host3.example.com:/etc/security/keytabs/pxf.service.keytab
- ```
-
- **Note:** Repeat this step for all of the keytab files that you created. Record the full path of the keytab file on each host machine. During installation, you will need to provide the path to finish configuring Kerberos for HAWQ.
-
-10. Change the ownership and permissions on the pxf.service.keytab files:
-
- ```
- $ ssh host1.example.com chown pxf:pxf /etc/security/keytabs/pxf.service.keytab
- $ ssh host1.example.com chmod 400 /etc/security/keytabs/pxf.service.keytab
- $ ssh host2.example.com chown pxf:pxf /etc/security/keytabs/pxf.service.keytab
- $ ssh host2.example.com chmod 400 /etc/security/keytabs/pxf.service.keytab
- $ ssh host3.example.com chown pxf:pxf /etc/security/keytabs/pxf.service.keytab
- $ ssh host3.example.com chmod 400 /etc/security/keytabs/pxf.service.keytab
- ```
-
- **Note:** Repeat this step for all of the keytab files that you copied.
-
-11. After you have created and copied all required keytab files to their respective hosts, continue to install the HAWQ and PXF services if necessary.
-
- If you already installed HAWQ and PXF and you enabled HDFS security *without* using Ambari, follow the steps in [\(Optional\) Enable Kerberos](#topic_iwy_5fb_1t) to finish enabling HAWQ and PXF security.
-
-12. On each PXF node, edit the /etc/gphd/pxf/conf/pxf-site.xml to specify the local keytab file and security principal. Add or uncomment the properties:
-
- ```
- <property>
- <name>pxf.service.kerberos.keytab</name>
- <value>/etc/security/phd/keytabs/pxf.service.keytab</value>
- <description>path to keytab file owned by pxf service
- with permissions 0400</description>
- </property>
-
- <property>
- <name>pxf.service.kerberos.principal</name>
- <value>pxf/_HOST@PHD.LOCAL</value>
- <description>Kerberos principal pxf service should use.
- _HOST is replaced automatically with hostnames
- FQDN</description>
- </property>
- ```
-
- Also uncomment the KDC section in yarn-client.xml if you want to configure YARN for security.
-
-13. Perform the remaining steps on the HAWQ master node as the `gpadmin` user:
- 1. Login to the HAWQ database master server as the `gpadmin` user:
-
- ```
- ssh hawq_master_fqdn
- ```
-
- 2. Run the following commands to set environment variables:
-
- ```
- $ source /usr/local/hawq/greenplum_path.sh
- ```
-
- **Note:** Substitute the correct value of MASTER\_DATA\_DIRECTORY for your configuration.
-
- 3. Run the following commands to enable security and configure the keytab file:
-
- ```
- $ hawq config -c enable_secure_filesystem -v ON
- $ hawq config -c krb_server_keyfile -v /etc/security/keytabs/hawq.service.keytab
- ```
-
- **Note:** Substitute the correct value of MASTER\_DATA\_DIRECTORY for your configuration.
-
- 4. Start the HAWQ service:
-
- ```
- $ hawq start cluster -a
- ```
-
- 5. Obtain a kerberos ticket and change the ownership and permissions of the HAWQ HDFS data directory:
-
- ```
- $ sudo -u hdfs kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs
- $ sudo -u hdfs hdfs dfs -chown -R gpadmin:gpadmin /hawq_data
- ```
-
- Substitute the actual HDFS data directory name for your system.
-
- 6. On the HAWQ master node and on all segment server nodes, edit the /usr/local/hawq/etc/hdfs-client.xml file to enable kerberos security and assign the HDFS NameNode principal. Add or uncomment the following properties in each file:
-
- ```
- <property>
- <name>hadoop.security.authentication</name>
- <value>kerberos</value>
- </property>
-
- ```
-
- 7. If you are using YARN for resource management, edit the yarn-client.xml file to enable kerberos security. Add or uncomment the following property in the yarn-client.xml file on each HAWQ node:
-
- ```
- <property>
- <name>hadoop.security.authentication</name>
- <value>kerberos</value>
- </property>
- ```
-
- 8. Restart HAWQ as the `gpadmin` user:
-
- ```
- $ hawq restart cluster -a -M fast
- ```
-
-
-## Validate the Installation <a id="topic_opr_3gp_15"></a>
-
-Perform these basic commands to ensure that the new cluster is functional.
-
-1. Ensure that you have initialized and started the new cluster by using the `hawq init cluster` command, as described in the previous procedures.
-
-1. Source the `greenplum_path.sh` file to set your environment for HAWQ. For RPM installations, enter:
-
- ```
- $ source /usr/local/hawq/greenplum_path.sh
- ```
-
- If you downloaded the tarball, substitute the path to the extracted `greenplum_path.sh` file \(for example `/opt/hawq-2.0.0.0/greenplum_path.sh`\).
-
-1. If you use a custom HAWQ master port number, set it in your environment. For example:
-
- ```
- $ export PGPORT=5432
- ```
-
-2. Start the `psql` interactive utility, connecting to the postgres database:
-
- ```
- $ psql -d postgres
- psql (8.2.15)
- Type "help" for help.
- postgres=#
- ```
-
-3. Create a new database and connect to it:
-
- ```
- postgres=# create database mytest;
- CREATE DATABASE
- postgres=# \c mytest
- You are now connected to database "mytest" as user "*username*".
- ```
-
-4. Create a new table and insert sample data:
-
- ```
- mytest=# create table t (i int);
- CREATE TABLE
- mytest=# insert into t select generate_series(1,100);
- ```
-
-5. Activate timing and perform a simple query:
-
- ```
- mytest=# \timing
- Timing is on.
- mytest=# select count(*) from t;
- count
- -------
- 100
- (1 row)
- Time: 7.266 ms
- ```
-
-
-## Install PXF Plugins <a id="topic_pnk_4cv_25"></a>
-
-If you plan on accessing data in external systems such as HDFS files, Hive or HBase:
-
-1. Follow the instructions in [Post-Install Procedure for Hive and HBase on HDP](/20/install/install-ambari.html#post-install-pxf) to complete the PXF configuration.
-
-2. Install the appropriate PXF plugin for the external system. We recommend installing the PXF plugin for the desired external system on all nodes in your cluster. See [Installing PXF Plugins](/200/hawq/pxf/InstallPXFPlugins.html).
diff --git a/install/install-components.md b/install/install-components.md
deleted file mode 100644
index d13353e..0000000
--- a/install/install-components.md
+++ /dev/null
@@ -1,395 +0,0 @@
----
-title: Installing Additional HAWQ Components
----
-
-This chapter describes how to install additional HAWQ components.
-
-- [Installing Cryptographic Functions for PostgreSQL](#)
- - [Install pgcrypto](#)
- - [Uninstalling pgcrypto](#)
-- [Installing PL/R](#)
- - [Install PL/R](#)
- - [Enable PL/R Language Support](#)
- - [Uninstalling PL/R](#)
-- [Installing PL/Java](#)
- - [Install the HAWQ PL/Java Extension](#)
- - [Enable PL/Java and Install JAR Files](#)
- - [Uninstalling PL/Java](#)
-- [Installing MADlib on HAWQ](#)
- - [Pre-requisites for Installing MADlib on HAWQ](#)
- - [Install MADlib on HAWQ](#)
-
-## Installing Cryptographic Functions for PostgreSQL <a id="installingcryptographicfunctionsforpostgresql"></a>
-
-pgcrypto is available as a package that you can download from the Pivotal Download Center and install using the Package Manager utility \(`gppkg`\). `gppkg` installs pgcrypto and other HAWQ extensions, along with any dependencies, on all hosts across a cluster. It will also automatically install extensions on new hosts in the case of system expansion and segment recovery.
-
-**Note:** Before you install the pgcrypto software package, make sure that your HAWQ database is running, you have sourced greenplum\_path.sh, and that the `$MASTER_DATA_DIRECTORY` and `$GPHOME` variables are set.
-
-### Install pgcrypto <a id="installpgcrypto"></a>
-
-Download the pgcrypto package from the Pivotal Download Center, then copy it to the master host. Install the software package by running the following command:
-
-```
-gppkg -i pgcrypto-1.0-rhel5-x86_64.gppkg
-```
-
-You will see output similar to the following.
-
-```
-[gpadmin@gp-single-host ~]$ gppkg -i pgcrypto-1.0-rhel5-x86_64.gppkg
-20120418:23:54:20:gppkg:gp-single-host:gpadmin-[INFO]:-Starting gppkg with args: -i pgcrypto-1.0-rhel5-x86_64.gppkg
-20120418:23:54:20:gppkg:gp-single-host:gpadmin-[INFO]:-Installing package pgcrypto-1.0-rhel5-x86_64.gppkg
-20120418:23:54:21:gppkg:gp-single-host:gpadmin-[INFO]:-Validating rpm installation cmdStr='rpm --test -i /usr/local/greenplum-db/./.tmp/pgcrypto-1.0-1.x86_64.rpm --dbpath /usr/local/greenplum-db/./share/packages/database --prefix /usr/local/greenplum-db/.'
-20120418:23:54:22:gppkg:gp-single-host:gpadmin-[INFO]:-Please run psql -d mydatabase -f $GPHOME/share/postgresql/contrib/pgcrypto.sql to enable the package.
-20120418:23:54:22:gppkg:gp-single-host:gpadmin-[INFO]:-pgcrypto-1.0-rhel5-x86_64.gppkg successfully installed.
-```
-
-### Uninstalling pgcrypto <a id="uninstallingpgcrypto"></a>
-
-#### Uninstall pgcrypto support
-
-To uninstall the pgcrypto objects, use uninstall\_pgcrypto.sql to remove pgcrypto support.
-
-For each database on which you enabled pgcrypto support, execute the following:
-
-```
-psql -d dbname -f $GPHOME/share/postgresql/contrib/uninstall_pgcrypto.sql
-```
-
-**Note:** This script does not remove dependent user-created objects.
-
-#### Uninstall the software package
-
-You can uninstall the pgcrypto software using the Greenplum Package Manager \(gppkg\), as follows:
-
-```
-gppkg -r pgcrypto-1.0
-```
-
-## Installing PL/R <a id="installingplr"></a>
-
-PL/R is available as a package that you can download from the Pivotal Download Center and install using the Package Manager utility \(`gppkg`\). `gppkg` installs PL/R and other Greenplum Database extensions, along with any dependencies, on all hosts across a cluster. It will also automatically install extensions on new hosts in the case of system expansion and segment recovery.
-
-**Note:** Before you install the PL/R software package, make sure that your HAWQ database is running, you have sourced greenplum\_path.sh, and that the `$MASTER_DATA_DIRECTORY` and `$GPHOME` variables are set.
-
-### Install PL/R <a id="installplr"></a>
-
-1. Download the PL/R package from the Pivotal Download Center, then copy it to the master host. Install the software package by running the following command:
-
- ```
- $ gppkg -i plr-1.0-rhel5-x86_64.gppkg
- ```
-
-2. Source the $GPHOME/greenplum\_path.sh file. The extension and the R environment are installed in the $GPHOME/ext/R-2.13.0/ directory.
-3. Restart the database:
-
- ```
- $ gpstop -r
- ```
-
-
-### Enable PL/R Language Support <a id="enablingplrlanguagesupport"></a>
-
-For each database that requires its use, register the PL/R language with the `CREATE LANGUAGE` SQL command or the `createlang` utility. For example, running the following command as the `gpadmin` user registers the language for a database named `testdb`:
-
-```
-$ createlang plr -d testdb
-```
-
-PL/R is registered as an untrusted language.
-
-You are now ready to create new PLR functions. A library of convenient PLR functions may be found in $GPHOME/share/postgresql/contrib/plr.sql. These functions may be installed by using the `psql` utility to execute plr.sql, as follows:
-
-```
-psql -d <dbname> -f $GPHOME/share/postgresql/contrib/plr.sql
-```
-
-### Uninstalling PL/R <a id="uninstallingplr"></a>
-
-When you remove PL/R language support from a database, the PL/R routines that you created in the database will no longer work.
-
-#### Remove PL/R Support for a Database <a id="removeplrsupportforadatabase"></a>
-
-For a database that no long requires the PL/R language, remove support for PL/R with the SQL command `DROP LANGUAGE`or the `droplang` utility. For example, running the following command as the `gpadmin` user removes support for PL/R from the database `testdb`:
-
-```
-$ droplang plr -d testdb
-```
-
-#### Uninstall the Software Package <a id="uninstallthesoftwarepackage"></a>
-
-If no databases have PL/R as a registered language, uninstall the Greenplum PL/R extension with the `gppkg` utility. This example uninstalls PL/R package version 1.0:
-
-```
-$ gppkg -r plr-1.0
-```
-
-You can run the `gppkg` utility with the options `-q --all` to list the installed extensions and their versions.
-
-Then, restart the database.
-
-```
-$ gpstop -r
-```
-
-##### Downloading and Installing R libraries
-
-For a given R library, identify all dependent R libraries and each library's web URL.This can be found by selecting the specific package from the following page: [http://cran.r-project.org/web/packages/available\_packages\_by\_name.html](http://cran.r-project.org/web/packages/available_packages_by_name.html)
-
-From the page for the arm library, you can see that this library requires the following R libraries:
-
-- Matrix
-- lattice
-- lme4
-- R2WinBUGS
-- coda
-- abind
-- foreign
-- MASS
-
-From the command line, use `wget` to download the tar.gz files for the required libraries to the master node:
-
-```
-$ wget http://cran.r-project.org/src/contrib/arm_1.5-03.tar.gz
-$ wget http://cran.r-project.org/src/contrib/Archive/Matrix/Matrix_1.0-1.tar.gz
-$ wget http://cran.r-project.org/src/contrib/Archive/lattice/lattice_0.19-33.tar.gz
-$ wget http://cran.r-project.org/src/contrib/lme4_0.999375-42.tar.gz
-$ wget http://cran.r-project.org/src/contrib/R2WinBUGS_2.1-18.tar.gz
-$ wget http://cran.r-project.org/src/contrib/coda_0.14-7.tar.gz
-$ wget http://cran.r-project.org/src/contrib/abind_1.4-0.tar.gz
-$ wget http://cran.r-project.org/src/contrib/foreign_0.8-49.tar.gz
-$ wget http://cran.r-project.org/src/contrib/MASS_7.3-17.tar.gz
-```
-
-Using `gpscp` and the hostname file, copy the tar.gz files to the same directory on all nodes of the HAWQ cluster. You may require `root` access to do this.
-
-```
-$ gpscp -f /home/gpadmin/hosts_all lattice_0.19-33.tar.gz =:/home/gpadmin
-$ gpscp -f /home/gpadmin/hosts_all Matrix_1.0-1.tar.gz =:/home/gpadmin
-$ gpscp -f /home/gpadmin/hosts_all abind_1.4-0.tar.gz =:/home/gpadmin
-$ gpscp -f /home/gpadmin/hosts_all coda_0.14-7.tar.gz =:/home/gpadmin
-$ gpscp -f /home/gpadmin/hosts_all R2WinBUGS_2.1-18.tar.gz =:/home/gpadmin
-$ gpscp -f /home/gpadmin/hosts_all lme4_0.999375-42.tar.gz =:/home/gpadmin
-$ gpscp -f /home/gpadmin/hosts_all MASS_7.3-17.tar.gz =:/home/gpadmin
-$ gpscp -f /home/gpadmin/hosts_all arm_1.5-03.tar.gz =:/home/gpadmin
-```
-
-Use `R CMD INSTALL` to install the packages from the command line. You may require `root` access to do this.
-
-```
-$ R CMD INSTALL lattice_0.19-33.tar.gz Matrix_1.0-1.tar.gz abind_1.4-0.tar.gz coda_0.14-7.tar.gz R2WinBUGS_2.1-18.tar.gz lme4_0.999375-42.tar.gz MASS_7.3-17.tar.gz arm_1.5-03.tar.gz
-```
-
-## Installing PL/Java <a id="installingpljava"></a>
-
-The PL/Java extension is available as a package that you can download from the Pivotal Download Center and then install with the Package Manager utility \(`gppkg`\).
-
-**Note:** Before you install PL/Java:
-
-- Ensure that the `$JAVA_HOME` variable is set to the same path on the master and all the segments.
-- Perform the following step on all machines to set up `ldconfig` for JDK:
-
- ```
- $ echo "$JAVA_HOME/jre/lib/amd64/server" > /etc/ld.so.conf.d/libjdk.conf
- $ ldconfig
- ```
-
-- If you are upgrading to the latest version of Java or installing it as part of the expansion process, follow the instructions in the chapter, *Expanding the HAWQ System* in the *HAWQ Administrator Guide*.
-- PL/Java is compatible with JDK 1.6 and 1.7.
-
-The `gppkg` utility installs HAWQ extensions, along with any dependencies, on all hosts across a cluster. It also automatically installs extensions on new hosts in the case of system expansion and segment recovery.
-
-To install and use PL/Java:
-
-1. Install the PL/Java extension.
-2. Enable the language for each database.
-3. Install user-created JAR files containing Java methods on all HAWQ hosts.
-4. Add the name of the JAR file to the HAWQ `pljava_classpath` environment variable. The variable lists the installed JAR files.
-
-**Note:** Before you install the PL/Java extension, make sure that your Greenplum database is running, you have sourced greenplum\_path.sh, and that the `$MASTER_DATA_DIRECTORY` and `$GPHOME` variables are set.
-
-### Install the HAWQ PL/Java Extension <a id="installingthehawqpljavaextension"></a>
-
-1. Download the PL/Java extension package from the Pivotal Download Center and copy it to the master host.
-2. Install the software extension package by running the `gppkg` command. This example installs the PL/Java extension package on a Linux system:
-
- ```
- $ gppkg -i pljava-1.1-rhel5-x86_64.gppkg
- ```
-
-3. Restart the database:
-
- ```
- $ gpstop -r
- ```
-
-4. Source the $GPHOME/greenplum\_path.sh file.
-
-### Enable PL/Java and Install JAR Files <a id="enablingpljavaandinstallingjarfiles"></a>
-
-Perform the following steps as the HAWQ administrator `gpadmin`:
-
-1. Enable PL/Java by running the $GPHOME/share/postgresql/pljava/install.sql SQL script in the databases that use PL/Java. For example, this example enables PL/Java on a database named `mytestdb`:
-
- ```
- $ psql -d mytestdb -f $GPHOME/share/postgresql/pljava/install.sql
- ```
-
- The install.sql script registers both the trusted and untrusted PL/Java.
-
-2. Copy your Java archives \(JAR files\) to $GPHOME/lib/postgresql/java/ on all the HAWQ hosts. This example uses the gpscp utility to copy the myclasses.jar file:
-
- ```
- $ gpscp -f gphosts_file myclasses.jar =:/usr/local/greenplum-db/lib/postgresql/java/
- ```
-
- The gphosts\_file file contains a list of the Greenplum Database hosts.
-
-3. Set the `pljava_classpath` server configuration parameter in the master hawq-site.xml file. The parameter value is a colon \(`:`\) separated list of the JAR files containing the Java classes used in any PL/Java functions. For example:
-
- ```
- $ gpconfig -c pljava_classpath -v \'examples.jar:myclasses.jar\' --masteronly
- ```
-
-4. Restart the database:
-
- ```
- $ gpstop -r
- ```
-
-5. **\(Optional\)** Pivotal provides an examples.sql file containing sample PL/Java functions that you can use for testing. Run the commands in this file to create the test functions \(which use the Java classes in examples.jar\):
-
- ```
- $ psql -f $GPHOME/share/postgresql/pljava/examples.sql
- ```
-
- Enabling the PL/Java extension in the `template1` database enables PL/Java in any new Greenplum databases:
-
- ```
- $ psql template1 -f $GPHOME/share/postgresql/pljava/install.sql
- ```
-
-
-#### Configuring PL/Java vmoptions <a id="configuringpljavavmoptions"></a>
-
-PL/Java JVM options can be configured via the `pljava_vmoptions` parameter in the hawq-site.xml file. For example, `pljava_vmoptions=-Xmx512M` sets the maximum heap size of the JVM. The default `Xmx` value is set to `-Xmx64M`.
-
-### Uninstalling PL/Java <a id="uninstallingpljava"></a>
-
-To uninstall PL/Java, you should:
-
-1. Remove PL/Java Support for a Database
-2. Uninstall the Java JAR files and Software Package
-
-#### Remove PL/Java Support for a Database <a id="removepljavasupportforadatabase"></a>
-
-For a database that no long requires the PL/Java language, remove support for PL/Java by running the uninstall.sql file as the `gpadmin` user. For example, the following command disables the PL/Java language in the specified database:
-
-```
-$ psql -d mydatabase -f $GPHOME/share/postgresql/pljava/uninstall.sql
-```
-
-#### Uninstall the Java JAR files and Software Package <a id="uninstallthejavajarfilesandsoftwarepackage"></a>
-
-If no databases have PL/Java as a registered language, remove the Java JAR files and uninstall the Greenplum PL/Java extension with the `gppkg` utility:
-
-1. Remove the `pljava_classpath` server configuration parameter in the master hawq-site.xml file.
-2. Remove the JAR files from the $GPHOME/lib/postgresql/java/ directory of the HAWQ hosts.
-3. Use the `gppkg` utility with the `-r` option to uninstall the PL/Java extension. The following example uninstalls the PL/Java extension on a Linux system:
-
- ```
- $ gppkg -r pljava-1.1
- ```
-
- You can run the `gppkg` utility with the options `-q --all` to list the installed extensions and their versions.
-
-4. After you uninstall the extension, restart the database:
-
- ```
- $ gpstop -r
- ```
-
-
-#### Installing Custom JARS <a id="installingcustomjars"></a>
-
-1. Copy the jar file on the master host in $GPHOME/lib/postgresql/java.
-2. Copy the jar file on all segments in the same location using `gpscp` from master:
-
- ```
- $ cd $GPHOME/lib/postgresql/java
- $ gpscp -f ~/hosts.txt myfunc.jar =:$GPHOME/lib/postgresql/java/
- ```
-
-3. Set `pljava_classpath` to include the newly-copied jar file:
- - From the `psql` session, execute set to affect the current session:
-
- ```
- set pljava_classpath='myfunc.jar';
- ```
-
- - To affect all sessions, use `gpconfig`:
-
- ```
- gpconfig -c pljava_classpath -v \'myfunc.jar\'
- ```
-
-
-## Installing MADlib on HAWQ <a id="installingmadlibonhawq1"></a>
-
-The MADlib library adds statistical and machine learning functionality to HAWQ. MADlib is provided as a package that you can download from the Pivotal Download Center and install using the Package Manager utility \(`gppkg`\). `gppkg` installs MADlib and other Greenplum Database extensions, along with any dependencies, on all hosts across a cluster. It also automatically installs extensions on new hosts in the case of system expansion and segment recovery.
-
-### Pre-requisites for Installing MADlib on HAWQ <a id="pre-requisitesforinstallingmadlibonhawq"></a>
-
-**Note:** Before you install the MADlib software package, make sure that your HAWQ database is running, that you have sourced greenplum\_path.sh, and that the `$MASTER_DATA_DIRECTORY` and `$GPHOME` variables are set.
-
-### Install MADlib on HAWQ <a id="installingmadlibonhawq2"></a>
-
-1. Download the MADlib package from the Pivotal Download Center, then copy it to the master host. Install the software package by running the command:
-
- ```
- $ gppkg -i madlib-ossv1.7.1_pv1.9.3_hawq1.3-rhel5-x86_64.gppkg
- ```
-
- The installation process begins and shows output similar to:
-
- ```
- 20150330:21:28:33:021734 gppkg:gpdb11:gpdbchina-[INFO]:-Starting
- gppkg with args: -i /data/home/gpdbchina/pulse2-data/agents/agent1/work/
- MADlib%20TINC%20Feature%20Test%20on%20HAWQ%201.3/rhel5_x86_64/madlib/
- madlib-ossv1.7.1_pv1.9.3_hawq1.3-rhel5-x86_64.gppkg
- 20150330:21:28:33:021734 gppkg:gpdb11:gpdbchina-[INFO]:-Installing package
- madlib-ossv1.7.1_pv1.9.3_hawq1.3-rhel5-x86_64.gppkgInstalled GPDB Version:
- pg_ctl (HAWQ) 1.3.0.0 build 12954
- [...]
- ```
-
-2. Restart the database:
-
- ```
- $ gpstop -r
- ```
-
-3. Source the $GPHOME/greenplum\_path.sh file.
-4. Deploy the MADlib objects to a database using the `GPHOME/madlib/bin/madpack` utility. The syntax for installing objects is:
-
- ```
- madpack install [-s schema\_name] -p hawq -c user@host:port/database
- ```
-
- The default schema name is `madlib`.
-
- For example, the following command install the objects to a database named "testdb" on server mdw:5432 using the gpadmin user:
-
- ```
- $ $GPHOME/madlib/bin/madpack install -s madlib -p hawq -c gpadmin@mdw:5432/testdb
- ```
-
- Enter the password for the specified user when prompted.
-
-5. To learn more about additional options for the `madpack` utility, enter:
-
- ```
- $GPHOME/madlib/bin/madpack --help
- ```
-
- See also the documentation available at [madlib.net](http://madlib.net/documentation/).
diff --git a/overview/ManagementTools.md b/overview/ManagementTools.md
index 48a1963..39f072b 100755
--- a/overview/ManagementTools.md
+++ b/overview/ManagementTools.md
@@ -6,4 +6,4 @@
The `hawq` command can init, start and stop each segment separately, and supports dynamic expansion of the cluster.
-See [HAWQ Management Tools Reference](/200/hawq/reference/cli/management_tools.html) for a list of all tools available in HAWQ.
+See [HAWQ Management Tools Reference](/20/reference/cli/management_tools.html) for a list of all tools available in HAWQ.
diff --git a/overview/ResourceManagement.md b/overview/ResourceManagement.md
index 0fb2f7e..5a5adda 100755
--- a/overview/ResourceManagement.md
+++ b/overview/ResourceManagement.md
@@ -11,4 +11,4 @@
- Dynamic resource allocation at query runtime. HAWQ dynamically allocates resources based on resource queue definitions. HAWQ automatically distributes resources based on running \(or queued\) queries and resource queue capacities.
- Resource limitations on virtual segments and queries. You can configure HAWQ to enforce limits on CPU and memory usage both for virtual segments and the resource queues used by queries.
-For more details on resource management in HAWQ and how it works, see [Managing Resources](/200/hawq/resourcemgmt/HAWQResourceManagement.html).
+For more details on resource management in HAWQ and how it works, see [Managing Resources](/20/resourcemgmt/HAWQResourceManagement.html).
diff --git a/overview/TableDistributionStorage.md b/overview/TableDistributionStorage.md
index ec179d2..bc41d92 100755
--- a/overview/TableDistributionStorage.md
+++ b/overview/TableDistributionStorage.md
@@ -16,7 +16,7 @@
On the other hand, for some queries, hash distributed tables are faster than randomly distributed tables. For example, hash distributed tables have some performance benefits for some TPC-H queries. You should choose the distribution policy that is best suited for your application's scenario.
-See [Choosing the Table Distribution Policy](/200/hawq/ddl/ddl-table.html) for more details.
+See [Choosing the Table Distribution Policy](/20/ddl/ddl-table.html) for more details.
## Data Locality
@@ -33,9 +33,9 @@
sources as readable or writable HAWQ tables. PXF has built-in connectors for
accessing data inside HDFS files, Hive tables, and HBase tables. PXF also
integrates with HCatalog to query Hive tables directly. See [Working with PXF
-and External Data](/200/hawq/pxf/PivotalExtensionFrameworkPXF.html) for more
+and External Data](/20/pxf/PivotalExtensionFrameworkPXF.html) for more
details.
Users can create custom PXF connectors to access other parallel data stores or
-processing engines. Connectors are Java plugins that use the PXF API. For more
-information see [PXF External Tables and API](/200/hawq/pxf/PXFExternalTableandAPIReference.html).
+processing engines. Connectors are Java plug-ins that use the PXF API. For more
+information see [PXF External Tables and API](/20/pxf/PXFExternalTableandAPIReference.html).
diff --git a/plext/using_pljava.md b/plext/using_pljava.md
index 861cfd6..c4e782b 100644
--- a/plext/using_pljava.md
+++ b/plext/using_pljava.md
@@ -2,7 +2,7 @@
title: Using PL/Java
---
-This section contains an overview of the HAWQ PL/Java language.
+This section contains an overview of the HAWQ PL/Java language.
## About PL/Java <a id="aboutpljava"></a>
@@ -57,7 +57,7 @@
The following server configuration parameters are used by PL/Java in HAWQ. These parameters replace the `pljava.*` parameters that are used in the standard PostgreSQL PL/Java implementation.
-<p class="note"><b>Note:</b> See the <a href="/200/hawq/reference/hawq-reference.html">HAWQ Reference</a> for information about HAWQ server configuration parameters.</p>
+<p class="note"><b>Note:</b> See the <a href="/20/reference/hawq-reference.html">HAWQ Reference</a> for information about HAWQ server configuration parameters.</p>
#### pljava\_classpath
@@ -113,7 +113,7 @@
Scalar types are mapped in a straightforward way. This table lists the current mappings.
-***Table: PL/Java data type mappings***
+***Table 1: PL/Java data type mappings***
| PostgreSQL | Java |
|------------|------|
@@ -378,6 +378,82 @@
}
}
```
+## Using JDBC <a id="usingjdbc"></a>
+
+PL/Java contains a JDBC driver that maps to the PostgreSQL SPI functions. A connection that maps to the current transaction can be obtained using the following statement:
+
+```java
+Connection conn =
+ DriverManager.getConnection("jdbc:default:connection");
+```
+
+After obtaining a connection, you can prepare and execute statements similar to other JDBC connections. These are limitations for the PL/Java JDBC driver:
+
+- The transaction cannot be managed in any way. Thus, you cannot use methods on the connection such as:
+ - `commit()`
+ - `rollback()`
+ - `setAutoCommit()`
+ - `setTransactionIsolation()`
+- Savepoints are available with some restrictions. A savepoint cannot outlive the function in which it was set and it must be rolled back or released by that same function.
+- A ResultSet returned from `executeQuery()` are always `FETCH_FORWARD` and `CONCUR_READ_ONLY`.
+- Meta-data is only available in PL/Java 1.1 or higher.
+- `CallableStatement` (for stored procedures) is not implemented.
+- The types `Clob` or `Blob` are not completely implemented, they need more work. The types `byte[]` and `String` can be used for `bytea` and `text` respectively.
+
+## Exception Handling <a id="exceptionhandling"></a>
+
+You can catch and handle an exception in the HAWQ backend just like any other exception. The backend `ErrorData` structure is exposed as a property in a class called `org.postgresql.pljava.ServerException` (derived from `java.sql.SQLException`) and the Java try/catch mechanism is synchronized with the backend mechanism.
+
+**Important:** You will not be able to continue executing backend functions until your function has returned and the error has been propagated when the backend has generated an exception unless you have used a savepoint. When a savepoint is rolled back, the exceptional condition is reset and you can continue your execution.
+
+## Savepoints <a id="savepoints"></a>
+
+HAWQ savepoints are exposed using the `java.sql.Connection` interface. Two restrictions apply.
+
+- A savepoint must be rolled back or released in the function where it was set.
+- A savepoint must not outlive the function where it was set.
+
+## Logging <a id="logging"></a>
+
+PL/Java uses the standard Java Logger. Hence, you can write things like:
+
+```java
+Logger.getAnonymousLogger().info( "Time is " + new
+Date(System.currentTimeMillis()));
+```
+
+At present, the logger uses a handler that maps the current state of the HAWQ configuration setting `log_min_messages` to a valid Logger level and that outputs all messages using the HAWQ backend function `elog()`.
+
+**Note:** The `log_min_messages` setting is read from the database the first time a PL/Java function in a session is executed. On the Java side, the setting does not change after the first PL/Java function execution in a specific session until the HAWQ session that is working with PL/Java is restarted.
+
+The following mapping apply between the Logger levels and the HAWQ backend levels.
+
+***Table 2: PL/Java Logging Levels Mappings***
+
+| java.util.logging.Level | HAWQ Level |
+|-------------------------|------------|
+| SEVERE ERROR | ERROR |
+| WARNING | WARNING |
+| CONFIG | LOG |
+| INFO | INFO |
+| FINE | DEBUG1 |
+| FINER | DEBUG2 |
+| FINEST | DEBUG3 |
+
+## Security <a id="security"></a>
+
+This section describes security aspects of using PL/Java.
+
+### Installation <a id="installation"></a>
+
+Only a database super user can install PL/Java. The PL/Java utility functions are installed using SECURITY DEFINER so that they execute with the access permissions that where granted to the creator of the functions.
+
+### Trusted Language <a id="trustedlang"></a>
+
+PL/Java is a trusted language. The trusted PL/Java language has no access to the file system as stipulated by PostgreSQL definition of a trusted language. Any database user can create and access functions in a trusted language.
+
+PL/Java also installs a language handler for the language `javau`. This version is not trusted and only a superuser can create new functions that use it. Any user can call the functions.
+
## Example <a id="pljavaexample"></a>
diff --git a/plext/using_plpython.md b/plext/using_plpython.md
new file mode 100644
index 0000000..b1a5bd6
--- /dev/null
+++ b/plext/using_plpython.md
@@ -0,0 +1,595 @@
+---
+title: Using PL/Python in HAWQ
+---
+
+This section contains an overview of the HAWQ PL/Python Language.
+
+## About HAWQ PL/Python <a id="abouthawqplpython"></a>
+
+PL/Python is a loadable procedural language. With the HAWQ PL/Python extension, you can write a HAWQ user-defined functions in Python that take advantage of Python features and modules to quickly build robust database applications.
+
+The HAWQ PL/Python extension is installed by default with HAWQ. HAWQ installs a version of Python and PL/Python. HAWQ uses the following Python installation:
+
+```shell
+$GPHOME/ext/python/
+```
+
+**Note:** HAWQ uses Python version 2.6.6.
+
+### HAWQ PL/Python Limitations <a id="hawqlimitations"></a>
+
+- HAWQ does not support PL/Python triggers.
+- PL/Python is available only as a HAWQ untrusted language.
+
+## Enabling and Removing PL/Python Support <a id="enableplpython"></a>
+
+The PL/Python language is installed with HAWQ. To create and run a PL/Python user-defined function (UDF) in a database, you must register the PL/Python language with the database.
+
+On every database where you want to install and enable PL/Python, connect to the database using the psql client.
+
+```shell
+$ psql -d <dbname>
+```
+
+Replace \<dbname\> with the name of the target database.
+
+Then, run the following SQL command:
+
+```shell
+psql# CREATE LANGUAGE plpythonu;
+```
+
+Note that `plpythonu` is installed as an “untrusted” language, meaning it does not offer any way of restricting what users can do in it.
+
+To remove support for plpythonu from a database, run the following SQL command:
+
+```shell
+psql# DROP LANGUAGE plpythonu;
+```
+
+## Developing Functions with PL/Python <a id="developfunctions"></a>
+
+The body of a PL/Python user-defined function is a Python script. When the function is called, its arguments are passed as elements of the array `args[]`. Named arguments are also passed as ordinary variables to the Python script. The result is returned from the PL/Python function with return statement, or yield statement in case of a result-set statement.
+
+The HAWQ PL/Python language module imports the Python module `plpy`. The module plpy implements these functions:
+
+- Functions to execute SQL queries and prepare execution plans for queries.
+ - `plpy.execute`
+ - `plpy.prepare`
+
+- Functions to manage errors and messages.
+ - `plpy.debug`
+ - `plpy.log`
+ - `plpy.info`
+ - `plpy.notice`
+ - `plpy.warning`
+ - `plpy.error`
+ - `plpy.fatal`
+ - `plpy.debug`
+
+## Executing and Preparing SQL Queries <a id="executepreparesql"></a>
+
+The PL/Python `plpy` module provides two Python functions to execute an SQL query and prepare an execution plan for a query, `plpy.execute` and `plpy.prepare`. Preparing the execution plan for a query is useful if you run the query from multiple Python functions.
+
+### plpy.execute <a id="plpyexecute"></a>
+
+Calling `plpy.execute` with a query string and an optional limit argument causes the query to be run and the result to be returned in a Python result object. The result object emulates a list or dictionary object. The rows returned in the result object can be accessed by row number and column name. The result set row numbering starts with 0 (zero). The result object can be modified. The result object has these additional methods:
+
+- `nrows` that returns the number of rows returned by the query.
+- `status` which is the `SPI_execute()` return value.
+
+For example, this Python statement in a PL/Python user-defined function executes a query.
+
+```python
+rv = plpy.execute("SELECT * FROM my_table", 5)
+```
+
+The `plpy.execute` function returns up to 5 rows from `my_table`. The result set is stored in the `rv` object. If `my_table` has a column `my_column`, it would be accessed as:
+
+```python
+my_col_data = rv[i]["my_column"]
+```
+
+Since the function returns a maximum of 5 rows, the index `i` can be an integer between 0 and 4.
+
+### plpy.prepare <a id="plpyprepare"></a>
+
+The function `plpy.prepare` prepares the execution plan for a query. It is called with a query string and a list of parameter types, if you have parameter references in the query. For example, this statement can be in a PL/Python user-defined function:
+
+```python
+plan = plpy.prepare("SELECT last_name FROM my_users WHERE
+ first_name = $1", [ "text" ])
+```
+
+The string text is the data type of the variable that is passed for the variable `$1`. After preparing a statement, you use the function `plpy.execute` to run it:
+
+```python
+rv = plpy.execute(plan, [ "Fred" ], 5)
+```
+
+The third argument is the limit for the number of rows returned and is optional.
+
+When you prepare an execution plan using the PL/Python module the plan is automatically saved. See the Postgres Server Programming Interface (SPI) documentation for information about the execution plans [http://www.postgresql.org/docs/8.2/static/spi.html](http://www.postgresql.org/docs/8.2/static/spi.html).
+
+To make effective use of saved plans across function calls you use one of the Python persistent storage dictionaries SD or GD.
+
+The global dictionary SD is available to store data between function calls. This variable is private static data. The global dictionary GD is public data, available to all Python functions within a session. Use GD with care.
+
+Each function gets its own execution environment in the Python interpreter, so that global data and function arguments from myfunc are not available to `myfunc2`. The exception is the data in the GD dictionary, as mentioned previously.
+
+This example uses the SD dictionary:
+
+```sql
+CREATE FUNCTION usesavedplan() RETURNS trigger AS $$
+ if SD.has_key("plan"):
+ plan = SD["plan"]
+ else:
+ plan = plpy.prepare("SELECT 1")
+ SD["plan"] = plan
+
+ # rest of function
+
+$$ LANGUAGE plpythonu;
+```
+
+## Handling Python Errors and Messages <a id="pythonerrors"></a>
+
+The message functions `plpy.error` and `plpy.fatal` raise a Python exception which, if uncaught, propagates out to the calling query, causing the current transaction or subtransaction to be aborted. The functions raise `plpy.ERROR(msg)` and raise `plpy.FATAL(msg)` are equivalent to calling `plpy.error` and `plpy.fatal`, respectively. The other message functions only generate messages of different priority levels.
+
+Whether messages of a particular priority are reported to the client, written to the server log, or both is controlled by the HAWQ server configuration parameters `log_min_messages` and `client_min_messages`. For information about the parameters, see the [Server Configuration Parameter Reference](../reference/HAWQSiteConfig.html).
+
+## Using the Dictionary GD to Improve PL/Python Performance <a id="dictionarygd"></a>
+
+In terms of performance, importing a Python module is an expensive operation and can affect performance. If you are importing the same module frequently, you can use Python global variables to load the module on the first invocation and not require importing the module on subsequent calls. The following PL/Python function uses the GD persistent storage dictionary to avoid importing a module if it has already been imported and is in the GD.
+
+```sql
+psql=#
+ CREATE FUNCTION pytest() returns text as $$
+ if 'mymodule' not in GD:
+ import mymodule
+ GD['mymodule'] = mymodule
+ return GD['mymodule'].sumd([1,2,3])
+$$;
+```
+
+## Installing Python Modules <a id="installpythonmodules"></a>
+
+When you install a Python module on HAWQ, the HAWQ Python environment must have the module added to it across all segment hosts in the cluster. When expanding HAWQ, you must add the Python modules to the new segment hosts. You can use the HAWQ utilities `hawq ssh` and `hawq scp` run commands on HAWQ hosts and copy files to the hosts. For information about the utilities, see the [HAWQ Management Tools Reference](../reference/cli/management_tools.html).
+
+As part of the HAWQ installation, the `gpadmin` user environment is configured to use Python that is installed with HAWQ.
+
+To check which Python is being used in your environment, use the `which` command:
+
+```bash
+$ which python
+```
+
+The command returns the location of the Python installation. The Python installed with HAWQ is in the HAWQ `ext/python` directory.
+
+```bash
+/$GPHOME/ext/python/bin/python
+```
+
+If you are building a Python module, you must ensure that the build creates the correct executable. For example on a Linux system, the build should create a 64-bit executable.
+
+Before building a Python module prior to installation, ensure that the appropriate software to build the module is installed and properly configured. The build environment is required only on the host where you build the module.
+
+These are examples of installing and testing Python modules:
+
+- Simple Python Module Installation Example (setuptools)
+- Complex Python Installation Example (NumPy)
+- Testing Installed Python Modules
+
+### Simple Python Module Installation Example (setuptools) <a id="simpleinstall"></a>
+
+This example manually installs the Python `setuptools` module from the Python Package Index repository. The module lets you easily download, build, install, upgrade, and uninstall Python packages.
+
+This example first builds the module from a package and installs the module on a single host. Then the module is built and installed on segment hosts.
+
+Get the module package from the Python Package Index site. For example, run this `wget` command on a HAWQ host as the gpadmin user to get the tar.gz file.
+
+```bash
+$ wget --no-check-certificate https://pypi.python.org/packages/source/s/setuptools/setuptools-18.4.tar.gz
+```
+
+Extract the files from the tar.gz file.
+
+```bash
+$ tar -xzvf setuptools-18.4.tar.gz
+```
+
+Go to the directory that contains the package files, and run the Python scripts to build and install the Python package.
+
+```bash
+$ cd setuptools-18.4
+$ python setup.py build && python setup.py install
+```
+
+The following Python command returns no errors if the module is available to Python.
+
+```bash
+$ python -c "import setuptools"
+```
+
+Copy the package to the HAWQ hosts with the `hawq scp` utility. For example, this command copies the tar.gz file from the current host to the host systems listed in the file `hawq-hosts`.
+
+```bash
+$ hawq scp -f hawq-hosts setuptools-18.4.tar.gz =:/home/gpadmin
+```
+
+Run the commands to build, install, and test the package with `hawq ssh` utility on the hosts listed in the file `hawq-hosts`. The file `hawq-hosts` lists all the remote HAWQ segment hosts:
+
+```bash
+$ hawq ssh -f hawq-hosts
+>>> tar -xzvf setuptools-18.4.tar.gz
+>>> cd setuptools-18.4
+>>> python setup.py build && python setup.py install
+>>> python -c "import setuptools"
+>>> exit
+```
+
+The `setuptools` package installs the `easy_install` utility that lets you install Python packages from the Python Package Index repository. For example, this command installs Python PIP utility from the Python Package Index site.
+
+```shell
+$ cd setuptools-18.4
+$ easy_install pip
+```
+
+You can use the `hawq ssh` utility to run the `easy_install` command on all the HAWQ segment hosts.
+
+### Complex Python Installation Example (NumPy) <a id="complexinstall"></a>
+
+This example builds and installs the Python module NumPy. NumPy is a module for scientific computing with Python. For information about NumPy, see [http://www.numpy.org/](http://www.numpy.org/).
+
+Building the NumPy package requires this software:
+- OpenBLAS libraries, an open source implementation of BLAS (Basic Linear Algebra Subprograms).
+- The gcc compilers: gcc, gcc-gfortran, and gcc-c++. The compilers are required to build the OpenBLAS libraries. See [OpenBLAS Prerequisites](#openblasprereq).
+
+This example process assumes `yum` is installed on all HAWQ segment hosts and the `gpadmin` user is a member of `sudoers` with `root` privileges on the hosts.
+
+Download the OpenBLAS and NumPy source files. For example, these `wget` commands download tar.gz files into the directory packages:
+
+```bash
+$ wget --directory-prefix=packages http://github.com/xianyi/OpenBLAS/tarball/v0.2.8
+$ wget --directory-prefix=packages http://sourceforge.net/projects/numpy/files/NumPy/1.8.0/numpy-1.8.0.tar.gz/download
+```
+
+Distribute the software to the HAWQ hosts. For example, if you download the software to `/home/gpadmin/packages`, these commands create the directory on the hosts and copies the software to hosts for the hosts listed in the `hawq-hosts` file.
+
+```bash
+$ hawq ssh -f hawq-hosts mkdir packages
+$ hawq scp -f hawq-hosts packages/* =:/home/gpadmin/packages
+```
+
+#### OpenBLAS Prerequisites <a id="openblasprereq"></a>
+
+1. If needed, use `yum` to install gcc compilers from system repositories. The compilers are required on all hosts where you compile OpenBLAS:
+
+ ```bash
+ $ sudo yum -y install gcc gcc-gfortran gcc-c++
+ ```
+
+ **Note:** If you cannot install the correct compiler versions with `yum`, you can download the gcc compilers, including gfortran, from source and install them.
+
+ These two commands download and install the compilers:
+
+ ```bash
+ $ wget http://gfortran.com/download/x86_64/snapshots/gcc-4.4.tar.xz
+ $ tar xf gcc-4.4.tar.xz -C /usr/local/
+ ```
+
+ If you installed `gcc` manually from a tar file, add the new `gcc` binaries to `PATH` and `LD_LIBRARY_PATH`:
+
+ ```bash
+ $ export PATH=$PATH:/usr/local/gcc-4.4/bin
+ $ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/gcc-4.4/lib
+ ```
+
+2. Create a symbolic link to `g++` and call it `gxx`:
+
+ ```bash
+ $ sudo ln -s /usr/bin/g++ /usr/bin/gxx
+ ```
+
+3. You might also need to create symbolic links to any libraries that have different versions available for example `libppl_c.so.4` to `libppl_c.so.2`.
+
+4. If needed, you can use the `hawq scp` utility to copy files to HAWQ hosts and the `hawq ssh` utility to run commands on the hosts.
+
+#### Build and Install OpenBLAS Libraries <a id="buildopenblas"></a>
+
+Before build and install the NumPy module, you install the OpenBLAS libraries. This section describes how to build and install the libraries on a single host.
+
+1. Extract the OpenBLAS files from the file. These commands extract the files from the OpenBLAS tar file and simplify the directory name that contains the OpenBLAS files.
+
+ ```bash
+ $ tar -xzf packages/v0.2.8 -C /home/gpadmin/packages
+ $ mv /home/gpadmin/packages/xianyi-OpenBLAS-9c51cdf /home/gpadmin/packages/ OpenBLAS
+ ```
+
+2. Compile OpenBLAS. These commands set the LIBRARY_PATH environment variable and run the make command to build OpenBLAS libraries.
+
+ ```bash
+ $ cd /home/gpadmin/packages/OpenBLAS
+ $ export LIBRARY_PATH=$LD_LIBRARY_PATH
+ $ make FC=gfortran USE_THREAD=0
+ ```
+
+3. Use these commands to install the OpenBLAS libraries in `/usr/local` as `root`, and then change the owner of the files to `gpadmin`.
+
+ ```bash
+ $ cd /home/gpadmin/packages/OpenBLAS/
+ $ sudo make PREFIX=/usr/local install
+ $ sudo ldconfig
+ $ sudo chown -R gpadmin /usr/local/lib
+ ```
+
+ The following libraries are installed, along with symbolic links:
+
+ ```bash
+ libopenblas.a -> libopenblas_sandybridge-r0.2.8.a
+ libopenblas_sandybridge-r0.2.8.a
+ libopenblas_sandybridge-r0.2.8.so
+ libopenblas.so -> libopenblas_sandybridge-r0.2.8.so
+ libopenblas.so.0 -> libopenblas_sandybridge-r0.2.8.so
+ ```
+
+4. You can use the `hawq ssh` utility to build and install the OpenBLAS libraries on multiple hosts.
+
+ All HAWQ hosts (master and segment hosts) have identical configurations. You can copy the OpenBLAS libraries from the system where they were built instead of building the OpenBlas libraries on all the hosts. For example, these `hawq ssh` and `hawq scp commands copy and install the OpenBLAS libraries on the hosts listed in the hawq-hosts file.
+
+```bash
+$ hawq ssh -f hawq-hosts -e 'sudo yum -y install gcc gcc-gfortran gcc-c++'
+$ hawq ssh -f hawq-hosts -e 'ln -s /usr/bin/g++ /usr/bin/gxx'
+$ hawq ssh -f hawq-hosts -e sudo chown gpadmin /usr/local/lib
+$ hawq scp -f hawq-hosts /usr/local/lib/libopen*sandy* =:/usr/local/lib
+```
+```bash
+$ hawq ssh -f hawq-hosts
+>>> cd /usr/local/lib
+>>> ln -s libopenblas_sandybridge-r0.2.8.a libopenblas.a
+>>> ln -s libopenblas_sandybridge-r0.2.8.so libopenblas.so
+>>> ln -s libopenblas_sandybridge-r0.2.8.so libopenblas.so.0
+>>> sudo ldconfig
+```
+
+#### Build and Install NumPy <a name="buildinstallnumpy"></a>
+
+After you have installed the OpenBLAS libraries, you can build and install NumPy module. These steps install the NumPy module on a single host. You can use the hawq ssh utility to build and install the NumPy module on multiple hosts.
+
+1. Go to the packages subdirectory and get the NumPy module source and extract the files.
+
+ ```bash
+ $ cd /home/gpadmin/packages
+ $ tar -xzf numpy-1.8.0.tar.gz
+ ```
+
+2. Set up the environment for building and installing NumPy.
+
+ ```bash
+ $ export BLAS=/usr/local/lib/libopenblas.a
+ $ export LAPACK=/usr/local/lib/libopenblas.a
+ $ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/
+ $ export LIBRARY_PATH=$LD_LIBRARY_PATH
+ ```
+
+3. Go to the NumPy directory and build and install NumPy. Building the NumPy package might take some time.
+
+ ```bash
+ $ cd numpy-1.8.0
+ $ python setup.py build
+ $ python setup.py install
+ ```
+
+ **Note:** If the NumPy module did not successfully build, the NumPy build process might need a site.cfg that specifies the location of the OpenBLAS libraries. Create the file `site.cfg` in the NumPy package directory:
+
+ ```bash
+ $ cd ~/packages/numpy-1.8.0
+ $ touch site.cfg
+ ```
+
+ Add the following to the `site.cfg` file and run the NumPy build command again:
+
+ <pre>
+ [default]
+ library_dirs = /usr/local/lib
+
+ [atlas]
+ atlas_libs = openblas
+ library_dirs = /usr/local/lib
+
+ [lapack]
+ lapack_libs = openblas
+ library_dirs = /usr/local/lib
+
+ # added for scikit-learn
+ [openblas]
+ libraries = openblas
+ library_dirs = /usr/local/lib
+ include_dirs = /usr/local/include
+ </pre>
+
+4. The following Python command ensures that the module is available for import by Python on a host system.
+
+ ```bash
+ $ python -c "import numpy"
+ ```
+
+5. Similar to the simple module installation, use the `hawq ssh` utility to build, install, and test the module on HAWQ segment hosts.
+
+5. The environment variables that are require to build the NumPy module are also required in the gpadmin user environment when running Python NumPy functions. You can use the `hawq ssh` utility with the `echo` command to add the environment variables to the `.bashrc` file. For example, these echo commands add the environment variables to the `.bashrc` file in the user home directory.
+
+ ```bash
+ $ echo -e '\n#Needed for NumPy' >> ~/.bashrc
+ $ echo -e 'export BLAS=/usr/local/lib/libopenblas.a' >> ~/.bashrc
+ $ echo -e 'export LAPACK=/usr/local/lib/libopenblas.a' >> ~/.bashrc
+ $ echo -e 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib' >> ~/.bashrc
+ $ echo -e 'export LIBRARY_PATH=$LD_LIBRARY_PATH' >> ~/.bashrc
+ ```
+
+## Testing Installed Python Modules <a id="testingpythonmodules"></a>
+
+You can create a simple PL/Python user-defined function (UDF) to validate that Python a module is available in HAWQ. This example tests the NumPy module.
+
+This PL/Python UDF imports the NumPy module. The function returns SUCCESS if the module is imported, and FAILURE if an import error occurs.
+
+```sql
+CREATE OR REPLACE FUNCTION plpy_test(x int)
+returns text
+as $$
+ try:
+ from numpy import *
+ return 'SUCCESS'
+ except ImportError, e:
+ return 'FAILURE'
+$$ language plpythonu;
+```
+
+Create a table that contains data on each HAWQ segment instance. Depending on the size of your HAWQ installation, you might need to generate more data to ensure data is distributed to all segment instances.
+
+```sql
+CREATE TABLE DIST AS (SELECT x FROM generate_series(1,50) x ) DISTRIBUTED RANDOMLY ;
+```
+
+This SELECT command runs the UDF on the segment hosts where data is stored in the primary segment instances.
+
+```sql
+SELECT gp_segment_id, plpy_test(x) AS status
+ FROM dist
+ GROUP BY gp_segment_id, status
+ ORDER BY gp_segment_id, status;
+```
+
+The SELECT command returns SUCCESS if the UDF imported the Python module on the HAWQ segment instance. If the SELECT command returns FAILURE, you can find the segment host of the segment instance host. The HAWQ system table `gp_segment_configuration` contains information about segment configuration. This command returns the host name for a segment ID.
+
+```sql
+SELECT hostname, content AS seg_ID FROM gp_segment_configuration
+ WHERE content = seg_id ;
+```
+
+If FAILURE is returned, these are some possible causes:
+
+- A problem accessing required libraries. For the NumPy example, HAWQ might have a problem accessing the OpenBLAS libraries or the Python libraries on a segment host.
+
+ Make sure you get no errors when running command on the segment host as the gpadmin user. This hawq ssh command tests importing the NumPy module on the segment host mdw1.
+
+ ```shell
+ $ hawq ssh -h mdw1 python -c "import numpy"
+ ```
+
+- If the Python import command does not return an error, environment variables might not be configured in the HAWQ environment. For example, the variables are not in the `.bashrc` file, or HAWQ might not have been restarted after adding the environment variables to the `.bashrc` file.
+
+ Ensure sure that the environment variables are properly set and then restart HAWQ. For the NumPy example, ensure the environment variables listed at the end of the section [Build and Install NumPy](#buildinstallnumpy) are defined in the `.bashrc` file for the gpadmin user on the master and segment hosts.
+
+ **Note:** On HAWQ master and segment hosts, the `.bashrc` file for the gpadmin user must source the file `$GPHOME/greenplum_path.sh`.
+
+## Examples <a id="examples"></a>
+
+This PL/Python UDF returns the maximum of two integers:
+
+```sql
+CREATE FUNCTION pymax (a integer, b integer)
+ RETURNS integer
+AS $$
+ if (a is None) or (b is None):
+ return None
+ if a > b:
+ return a
+ return b
+$$ LANGUAGE plpythonu;
+```
+
+You can use the STRICT property to perform the null handling instead of using the two conditional statements.
+
+```sql
+CREATE FUNCTION pymax (a integer, b integer)
+ RETURNS integer AS $$
+return max(a,b)
+$$ LANGUAGE plpythonu STRICT ;
+```
+
+You can run the user-defined function pymax with SELECT command. This example runs the UDF and shows the output.
+
+```sql
+SELECT ( pymax(123, 43));
+column1
+---------
+ 123
+(1 row)
+```
+
+This example that returns data from an SQL query that is run against a table. These two commands create a simple table and add data to the table.
+
+```sql
+CREATE TABLE sales (id int, year int, qtr int, day int, region text)
+ DISTRIBUTED BY (id) ;
+
+INSERT INTO sales VALUES
+ (1, 2014, 1,1, 'usa'),
+ (2, 2002, 2,2, 'europe'),
+ (3, 2014, 3,3, 'asia'),
+ (4, 2014, 4,4, 'usa'),
+ (5, 2014, 1,5, 'europe'),
+ (6, 2014, 2,6, 'asia'),
+ (7, 2002, 3,7, 'usa') ;
+```
+
+This PL/Python UDF executes a SELECT command that returns 5 rows from the table. The Python function returns the REGION value from the row specified by the input value. In the Python function, the row numbering starts from 0. Valid input for the function is an integer between 0 and 4.
+
+```sql
+CREATE OR REPLACE FUNCTION mypytest(a integer)
+ RETURNS text
+AS $$
+ rv = plpy.execute("SELECT * FROM sales ORDER BY id", 5)
+ region = rv[a]["region"]
+ return region
+$$ language plpythonu;
+```
+
+Running this SELECT statement returns the REGION column value from the third row of the result set.
+
+```sql
+SELECT mypytest(2) ;
+```
+
+This command deletes the UDF from the database.
+
+```sql
+DROP FUNCTION mypytest(integer) ;
+```
+
+## References <a id="references"></a>
+
+This section lists references for using PL/Python.
+
+### Technical References <a id="technicalreferences"></a>
+
+For information about PL/Python see the PostgreSQL documentation at [http://www.postgresql.org/docs/8.2/static/plpython.html](http://www.postgresql.org/docs/8.2/static/plpython.html).
+
+For information about Python Package Index (PyPI), see [https://pypi.python.org/pypi](https://pypi.python.org/pypi).
+
+These are some Python modules that can be downloaded:
+
+- SciPy library provides user-friendly and efficient numerical routines such as routines for numerical integration and optimization [http://www.scipy.org/scipylib/index.html](http://www.scipy.org/scipylib/index.html). This wget command downloads the SciPy package tar file.
+
+ ```shell
+$ wget http://sourceforge.net/projects/scipy/files/scipy/0.10.1/ scipy-0.10.1.tar.gz/download
+```
+
+- Natural Language Toolkit (nltk) is a platform for building Python programs to work with human language data http://www.nltk.org/. This wget command downloads the nltk package tar file.
+
+ ```shell
+$ wget http://pypi.python.org/packages/source/n/nltk/nltk-2.0.2.tar.gz#md5=6e714ff74c3398e88be084748df4e657
+ ```
+
+ **Note:** The Python package Distribute [https://pypi.python.org/pypi/](https://pypi.python.org/pypi/) distribute is required for `nltk`. The Distribute module should be installed before the `ntlk` package. This wget command downloads the Distribute package tar file.
+
+```shell
+$ wget http://pypi.python.org/packages/source/d/distribute/distribute-0.6.21.tar.gz
+```
+
+### Useful Reading <a id="usefulreading"></a>
+
+For information about the Python language, see [http://www.python.org/](http://www.python.org/).
+
+A set of slides that were used in a talk about how the Pivotal Data Science team uses the PyData stack in the Pivotal MPP databases and on Pivotal Cloud Foundry [http://www.slideshare.net/SrivatsanRamanujam/all-thingspythonpivotal](http://www.slideshare.net/SrivatsanRamanujam/all-thingspythonpivotal).
+
diff --git a/plext/using_plr.md b/plext/using_plr.md
index f6a19f6..b6fec4c 100644
--- a/plext/using_plr.md
+++ b/plext/using_plr.md
@@ -40,7 +40,11 @@
```sql
CREATE TYPE t1 AS ...
-Use this TYPE when defining your PL/R function
+```
+
+Use this TYPE when defining your PL/R function:
+
+```sql
... RETURNS SET OF t1 AS ...
```
@@ -193,7 +197,7 @@
[http://www.r-project.org/](http://www.r-project.org/) - The R Project home page
-[https://github.com/pivotalsoftware/gp-r](https://github.com/pivotalsoftware/gp-r) - GitHub repository that contains information about using R with Pivotal software including Pivotal HDB.
+[https://github.com/pivotalsoftware/gp-r](https://github.com/pivotalsoftware/gp-r) - GitHub repository that contains information about using R.
[https://github.com/pivotalsoftware/PivotalR](https://github.com/pivotalsoftware/PivotalR) - GitHub repository for PivotalR, a package that provides an R interface to operate on HAWQ tables and views that is similar to the R `data.frame`. PivotalR also supports using the machine learning package MADlib directly from R.
diff --git a/pxf/ConfigurePXF.html.md.erb b/pxf/ConfigurePXF.html.md.erb
new file mode 100644
index 0000000..d07b00f
--- /dev/null
+++ b/pxf/ConfigurePXF.html.md.erb
@@ -0,0 +1,67 @@
+---
+title: Configuring PXF
+---
+
+This topic describes how to configure the PXF service.
+
+**Note:** After you make any changes to a PXF configuration file (such as `pxf-profiles.xml` for adding custom profiles), propagate the changes to all nodes with PXF installed, and then restart the PXF service on all nodes.
+
+## Setting up the Java Classpath<a id="settingupthejavaclasspath"></a>
+
+The classpath for the PXF service is set during the plug-in installation process. Administrators should only modify it when adding new PXF connectors. The classpath is defined in two files:
+
+1. `/etc/pxf/conf/pxf-private.classpath` – contains all the required resources to run the service, including pxf-hdfs, pxf-hbase, and pxf-hive plug-ins. This file must not be edited or removed.
+2. `/etc/pxf/conf/pxf-public.classpath` – user defined resources (such as custom profiles) can be added here. For example, specify resources for running a user-defined plug-in. The classpath resources should be defined one per line. Wildcard characters can be used in the name of the resource, but not in the full path. See [Adding and Updating Profiles](ReadWritePXF.html#addingandupdatingprofiles) for information on adding custom profiles.
+
+After changing the classpath files, the PXF service must be restarted.
+
+## Setting up the JVM Command Line Options for the PXF Service<a id="settingupthejvmcommandlineoptionsforpxfservice"></a>
+
+The PXF service JVM command line options can be added or modified for each pxf-service instance in the `/var/pxf/pxf-service/bin/setenv.sh` file:
+
+Currently the `JVM_OPTS` parameter is set with the following values for maximum Java heap size and thread stack size:
+
+``` shell
+JVM_OPTS="-Xmx512M -Xss256K"
+```
+
+After adding or modifying the JVM command line options, the PXF service must be restarted.
+
+## Using PXF on a Secure HDFS Cluster<a id="topic_i3f_hvm_ss"></a>
+
+You can use PXF on a secure HDFS cluster. Read, write, and analyze operations for PXF tables on HDFS files are enabled. It requires no changes to preexisting PXF tables from a previous version.
+
+### Requirements<a id="requirements"></a>
+
+- Both HDFS and YARN principals are created and are properly configured.
+- HAWQ is correctly configured to work in secure mode.
+
+Please refer to [Troubleshooting PXF](TroubleshootingPXF.html) for common errors related to PXF security and their meaning.
+
+## Credentials for Remote Services<a id="credentialsforremoteservices"></a>
+
+Credentials for remote services allows a PXF plug-in to access a remote service that requires credentials.
+
+### In HAWQ<a id="inhawq"></a>
+
+Two parameters for credentials are implemented in HAWQ:
+
+- `pxf_remote_service_login` – a string of characters detailing information regarding login (i.e. user name).
+- `pxf_remote_service_secret` – a string of characters detailing information that is considered secret (i.e. password).
+
+Currently, the contents of the two parameters are stored in memory, without any security, for the whole session. Leaving the session will insecurely drop the contents of the parameters.
+
+**Important:** These parameters are temporary and could soon be deprecated, in favor of a complete solution for managing credentials for remote services in PXF.
+
+### In a PXF Plug-in<a id="inapxfplugin"></a>
+
+In a PXF plug-in, the contents of the two credentials parameters is available through the following InputData API functions:
+
+``` java
+string getLogin()
+string getSecret()
+```
+
+Both functions return 'null' if the corresponding HAWQ parameter was set to an empty string or was not set at all.
+
+
diff --git a/pxf/HBasePXF.html.md.erb b/pxf/HBasePXF.html.md.erb
new file mode 100644
index 0000000..ab8b1d9
--- /dev/null
+++ b/pxf/HBasePXF.html.md.erb
@@ -0,0 +1,105 @@
+---
+title: Accessing HBase Data
+---
+
+## Prerequisites<a id="installingthepxfhbaseplugin"></a>
+
+Before trying to access HBase data with PXF, verify the following:
+
+- The `/etc/hbase/conf/hbase-env.sh` configuration file must reference the `pxf-hbase.jar`. For example, `/etc/hbase/conf/hbase-env.sh` should include the line:
+
+ ``` bash
+ export HBASE_CLASSPATH=${HBASE_CLASSPATH}:/usr/lib/pxf/pxf-hbase.jar
+ ```
+
+ **Note:** You must restart HBase after making any changes to the HBase configuration.
+
+- PXF HBase plug-in is installed on all cluster nodes.
+- HBase and ZooKeeper jars are installed on all cluster nodes.
+
+## Syntax<a id="syntax3"></a>
+
+To create an external HBase table, use the following syntax:
+
+``` sql
+CREATE [READABLE|WRITABLE] EXTERNAL TABLE table_name
+ ( column_name data_type [, ...] | LIKE other_table )
+LOCATION ('pxf://namenode[:port]/hbase-table-name?Profile=HBase')
+FORMAT 'CUSTOM' (Formatter='pxfwritable_import');
+```
+
+The HBase profile is equivalent to the following PXF parameters:
+
+- Fragmenter=org.apache.hawq.pxf.plugins.hbase.HBaseDataFragmenter
+- Accessor=org.apache.hawq.pxf.plugins.hbase.HBaseAccessor
+- Resolver=org.apache.hawq.pxf.plugins.hbase.HBaseResolver
+
+## Column Mapping<a id="columnmapping"></a>
+
+Most HAWQ external tables (PXF or others) require that the HAWQ table attributes match the source data record layout, and include all the available attributes. With HAWQ, however, you use the PXF HBase plug-in to specify the subset of HBase qualifiers that define the HAWQ PXF table. To set up a clear mapping between each attribute in the PXF table and a specific qualifier in the HBase table, you can use either direct mapping or indirect mapping. In addition, the HBase row key is handled in a special way.
+
+### Row Key<a id="rowkey"></a>
+
+You can use the HBase table row key in several ways. For example, you can see them using query results, or you can run a WHERE clause filter on a range of row key values. To use the row key in the HAWQ query, define the HAWQ table with the reserved PXF attribute `recordkey.` This attribute name tells PXF to return the record key in any key-value based system and in HBase.
+
+**Note:** Because HBase is byte and not character-based, you should define the recordkey as type bytea. This may result in better ability to filter data and increase performance.
+
+``` sql
+CREATE EXTERNAL TABLE <tname> (recordkey bytea, ... ) LOCATION ('pxf:// ...')
+```
+
+### Direct Mapping<a id="directmapping"></a>
+
+Use Direct Mapping to map HAWQ table attributes to HBase qualifiers. You can specify the HBase qualifier names of interest, with column family names included, as quoted values.
+
+For example, you have defined an HBase table called `hbase_sales` with multiple column families and many qualifiers. To create a HAWQ table with these attributes:
+
+- `rowkey`
+- qualifier `saleid` in the column family `cf1`
+- qualifier `comments` in the column family `cf8`
+
+use the following `CREATE EXTERNAL TABLE` syntax:
+
+``` sql
+CREATE EXTERNAL TABLE hbase_sales (
+ recordkey bytea,
+ "cf1:saleid" int,
+ "cf8:comments" varchar
+) ...
+```
+
+The PXF HBase plug-in uses these attribute names as-is and returns the values of these HBase qualifiers.
+
+### Indirect Mapping (via Lookup Table)<a id="indirectmappingvialookuptable"></a>
+
+The direct mapping method is fast and intuitive, but using indirect mapping helps to reconcile HBase qualifier names with HAWQ behavior:
+
+- HBase qualifier names may be longer than 32 characters. HAWQ has a 32-character limit on attribute name size.
+- HBase qualifier names can be binary or non-printable. HAWQ attribute names are character based.
+
+In either case, Indirect Mapping uses a lookup table on HBase. You can create the lookup table to store all necessary lookup information. This works as a template for any future queries. The name of the lookup table must be `pxflookup` and must include the column family named `mapping`.
+
+Using the sales example in Direct Mapping, if our `rowkey` represents the HBase table name and the `mapping` column family includes the actual attribute mapping in the key value form of`<hawq attr name>=<hbase cf:qualifier>`.
+
+#### Example<a id="example5"></a>
+
+This example maps the `saleid` qualifier in the `cf1` column family to the HAWQ `id` column and the `comments` qualifier in the `cf8` family to the HAWQ `cmts` column.
+
+| (row key) | mapping |
+|-----------|-------------------|
+| sales | id=cf1:saleid |
+| sales | cmts=cf8:comments |
+
+The mapping assigned new names for each qualifier. You can use these names in your HAWQ table definition:
+
+``` sql
+CREATE EXTERNAL TABLE hbase_sales (
+ recordkey bytea
+ id int,
+ cmts varchar
+) ...
+```
+
+PXF automatically matches HAWQ to HBase column names when a `pxflookup` table exists in HBase.
+
+
diff --git a/pxf/HDFSFileDataPXF.html.md.erb b/pxf/HDFSFileDataPXF.html.md.erb
new file mode 100644
index 0000000..a162f0b
--- /dev/null
+++ b/pxf/HDFSFileDataPXF.html.md.erb
@@ -0,0 +1,507 @@
+---
+title: Accessing HDFS File Data
+---
+
+## Prerequisites<a id="installingthepxfhdfsplugin"></a>
+
+Before working with HDFS file data using HAWQ and PXF, you should perform the following operations:
+
+- Test PXF on HDFS before connecting to Hive or HBase.
+- Ensure that all HDFS users have read permissions to HDFS services and that write permissions have been limited to specific users.
+
+## Syntax<a id="syntax1"></a>
+
+The syntax for creating an external HDFS file is as follows:
+
+``` sql
+CREATE [READABLE|WRITABLE] EXTERNAL TABLE table_name
+ ( column_name data_type [, ...] | LIKE other_table )
+LOCATION ('pxf://host[:port]/path-to-data?<pxf parameters>[&custom-option=value...]')
+ FORMAT '[TEXT | CSV | CUSTOM]' (<formatting_properties>);
+```
+
+where `<pxf parameters>` is:
+
+``` pre
+ FRAGMENTER=fragmenter_class&ACCESSOR=accessor_class&RESOLVER=resolver_class]
+ | PROFILE=profile-name
+```
+
+**Note:** Omit the `FRAGMENTER` parameter for `READABLE` external tables.
+
+Use an SQL `SELECT` statement to read from an HDFS READABLE table:
+
+``` sql
+SELECT ... FROM table_name;
+```
+
+Use an SQL `INSERT` statement to add data to an HDFS WRITABLE table:
+
+``` sql
+INSERT INTO table_name ...;
+```
+
+To read the data in the files or to write based on the existing format, use `FORMAT`, `PROFILE`, or one of the classes.
+
+This topic describes the following:
+
+- FORMAT clause
+- Profile
+- Accessor
+- Resolver
+- Avro
+
+**Note:** For more details about the API and classes, see [PXF External Tables and API](PXFExternalTableandAPIReference.html#pxfexternaltableandapireference).
+
+### FORMAT clause<a id="formatclause"></a>
+
+Use one of the following formats to read data with any PXF connector:
+
+- `FORMAT 'TEXT'`: Use with plain delimited text files on HDFS.
+- `FORMAT 'CSV'`: Use with comma-separated value files on HDFS.
+- `FORMAT 'CUSTOM'`: Use with all other files, including Avro format and binary formats. Must always be used with the built-in formatter '`pxfwritable_import`' (for read) or '`pxfwritable_export`' (for write).
+
+**Note:** When creating PXF external tables, you cannot use the `HEADER` option in your `FORMAT` specification.
+
+### Profile<a id="topic_ab2_sxy_bv"></a>
+
+For plain or comma-separated text files in HDFS use either the `HdfsTextSimple` or `HdfsTextMulti` Profile, or the classname org.apache.hawq.pxf.plugins.hdfs.*HdfsDataFragmenter*. Use the `Avro` profile for Avro files. See [Using Profiles to Read and Write Data](ReadWritePXF.html#readingandwritingdatawithpxf) for more information.
+
+**Note:** For read tables, you must include a Profile or a Fragmenter in the table definition.
+
+### Accessor<a id="accessor"></a>
+
+The choice of an Accessor depends on the HDFS data file type.
+
+**Note:** You must include either a Profile or an Accessor in the table definition.
+
+<table>
+<colgroup>
+<col width="25%" />
+<col width="25%" />
+<col width="25%" />
+<col width="25%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>File Type</th>
+<th>Accessor</th>
+<th>FORMAT clause</th>
+<th>Comments</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>Plain Text delimited</td>
+<td>org.apache.hawq.pxf.plugins. hdfs.LineBreakAccessor</td>
+<td>FORMAT 'TEXT' (<em>format param list</em>)</td>
+<td> Read + Write
+<p>You cannot use the <code class="ph codeph">HEADER</code> option.</p></td>
+</tr>
+<tr class="even">
+<td>Plain Text CSV </td>
+<td>org.apache.hawq.pxf.plugins. hdfs.LineBreakAccessor</td>
+<td>FORMAT 'CSV' (<em>format param list</em>) </td>
+<td><p>LineBreakAccessor is parallel and faster.</p>
+<p>Use if each logical data row is a physical data line.</p>
+<p>Read + Write </p>
+<p>You cannot use the <code class="ph codeph">HEADER</code> option.</p></td>
+</tr>
+<tr class="odd">
+<td>Plain Text CSV </td>
+<td>org.apache.hawq.pxf.plugins. hdfs.QuotedLineBreakAccessor</td>
+<td>FORMAT 'CSV' (<em>format param list</em>) </td>
+<td><p>QuotedLineBreakAccessor is slower and non-parallel.</p>
+<p>Use if the data includes embedded (quoted) linefeed characters.</p>
+<p>Read Only </p>
+<p>You cannot use the <code class="ph codeph">HEADER</code> option.</p></td>
+</tr>
+<tr class="even">
+<td>SequenceFile</td>
+<td>org.apache.hawq.pxf.plugins. hdfs.SequenceFileAccessor</td>
+<td>FORMAT 'CUSTOM' (formatter='pxfwritable_import')</td>
+<td> Read + Write (use formatter='pxfwritable_export' for write)</td>
+</tr>
+<tr class="odd">
+<td>AvroFile</td>
+<td>org.apache.hawq.pxf.plugins. hdfs.AvroFileAccessor</td>
+<td>FORMAT 'CUSTOM' (formatter='pxfwritable_import')</td>
+<td> Read Only</td>
+</tr>
+</tbody>
+</table>
+
+### Resolver<a id="resolver"></a>
+
+Choose the Resolver format if data records are serialized in the HDFS file.
+
+**Note:** You must include a Profile or a Resolver in the table definition.
+
+<table>
+<colgroup>
+<col width="33%" />
+<col width="33%" />
+<col width="33%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Record Serialization</th>
+<th>Resolver</th>
+<th>Comments</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>Avro</td>
+<td>org.apache.hawq.pxf.plugins. hdfs.AvroResolver</td>
+<td><ul>
+<li>Avro files include the record schema, Avro serialization can be used in other file types (e.g, Sequence File). </li>
+<li>For Avro serialized records outside of an Avro file, include a schema file name (.avsc) in the url under the optional <code class="ph codeph">Schema-Data </code>option.</li>
+<li>Deserialize Only (Read) .</li>
+</ul></td>
+</tr>
+<tr class="even">
+<td>Java Writable</td>
+<td>org.apache.hawq.pxf.plugins. hdfs.WritableResolver</td>
+<td><ul>
+<li>Include the name of the Java class that uses Writable serialization in the URL under the optional <code class="ph codeph">Schema-Data.</code></li>
+<li>The class file must exist in the public stage directory (or in Hadoop's class path).</li>
+<li>Deserialize and Serialize (Read + Write). </li>
+<li>See <a href="#customizedwritableschemafileguidelines">Customized Writable Schema File Guidelines</a>.</li>
+</ul></td>
+</tr>
+<tr class="odd">
+<td>None (plain text)</td>
+<td>org.apache.hawq.pxf.plugins. hdfs.StringPassResolver</td>
+<td><ul>
+<li>Does not serialize plain text records. The database parses plain records. Passes records as they are.</li>
+<li>Deserialize and Serialize (Read + Write).</li>
+</ul></td>
+</tr>
+</tbody>
+</table>
+
+#### Schema File Guidelines for WritableResolver<a id="customizedwritableschemafileguidelines"></a>
+
+When using a WritableResolver, a schema file needs to be defined. The file needs to be a Java class file and must be on the class path of PXF.
+
+The class file must follow the following requirements:
+
+1. Must implement org.apache.hadoop.io.Writable interface.
+2. WritableResolver uses reflection to recreate the schema and populate its fields (for both read and write). Then it uses the Writable interface functions to read/write. Therefore, fields must be public, to enable access to them. Private fields will be ignored.
+3. Fields are accessed and populated in the order in which they are declared in the class file.
+4. Supported field types:
+ - boolean
+ - byte array
+ - double
+ - float
+ - int
+ - long
+ - short
+ - string
+
+ Arrays of any of the above types are supported, but the constructor must define the array size so the reflection will work.
+
+### Additional Options<a id="additionaloptions"></a>
+
+<a id="additionaloptions__table_skq_kpz_4p"></a>
+
+<table>
+<caption><span class="tablecap">Table 1. Additional PXF Options</span></caption>
+<colgroup>
+<col width="50%" />
+<col width="50%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Option Name</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>COLLECTION_DELIM</td>
+<td>(Avro or Hive profiles only.) The delimiter character(s) to place between entries in a top-level array, map, or record field when PXF maps a Hive or Avro complex data type to a text column. The default is a "," character.</td>
+</tr>
+<tr class="even">
+<td>COMPRESSION_CODEC</td>
+<td><ul>
+<li>Useful for WRITABLE PXF tables.</li>
+<li>Specifies the compression codec class name for compressing the written data. The class must implement the org.apache.hadoop.io.compress.CompressionCodec interface.</li>
+<li> Some valid values are org.apache.hadoop.io.compress.DefaultCodec org.apache.hadoop.io.compress.GzipCodec org.apache.hadoop.io.compress.BZip2Codec.</li>
+<li>Note: org.apache.hadoop.io.compress.BZip2Codec runs in a single thread and can be slow.</li>
+<li>This option has no default value. </li>
+<li>When the option is not defined, no compression will be done.</li>
+</ul></td>
+</tr>
+<tr class="odd">
+<td>COMPRESSION_TYPE</td>
+<td><ul>
+<li>Useful WRITABLE PXF tables with SequenceFileAccessor.</li>
+<li>Ignored when COMPRESSION_CODEC is not defined.</li>
+<li>Specifies the compression type for sequence file.</li>
+<li>Valid options are:
+<ul>
+<li>RECORD - only the value part of each row is compressed.</li>
+<li>BLOCK - both keys and values are collected in 'blocks' separately and compressed.</li>
+</ul></li>
+<li>Default value: RECORD.</li>
+</ul></td>
+</tr>
+<tr class="even">
+<td>MAPKEY_DELIM</td>
+<td>(Avro or Hive profiles only.) The delimiter character(s) to place between the key and value of a map entry when PXF maps a Hive or Avro complex data type to a text colum. The default is a ":" character.</td>
+</tr>
+<tr class="odd">
+<td>RECORDKEY_DELIM</td>
+<td>(Avro profile only.) The delimiter character(s) to place between the field name and value of a record entry when PXF maps an Avro complex data type to a text colum. The default is a ":" character.</td>
+</tr>
+<tr class="even">
+<td>SCHEMA-DATA</td>
+<td>The data schema file used to create and read the HDFS file. For example, you could create an avsc (for Avro), or a Java class (for Writable Serialization) file. Make sure that you have added any JAR files containing the schema to <code class="ph codeph">pxf-public.classpath</code>.
+<p>This option has no default value.</p></td>
+</tr>
+<tr class="odd">
+<td>THREAD-SAFE</td>
+<td>Determines if the table query can run in multithread mode or not. When set to FALSE, requests will be handled in a single thread.
+<p>Should be set when a plug-in or other elements that are not thread safe are used (e.g. compression codec).</p>
+<p>Allowed values: TRUE, FALSE. Default value is TRUE - requests can run in multithread mode.</p></td>
+</tr>
+<tr class="even">
+<td> <custom></td>
+<td>Any option added to the pxf URI string will be accepted and passed, along with its value, to the Fragmenter, Accessor, and Resolver implementations.</td>
+</tr>
+</tbody>
+</table>
+
+## Accessing Data on a High Availability HDFS Cluster<a id="accessingdataonahighavailabilityhdfscluster"></a>
+
+To access data on a High Availability HDFS cluster, change the authority in the URI in the LOCATION. Use *HA\_nameservice* instead of *name\_node\_host:51200*.
+
+``` sql
+CREATE [READABLE|WRITABLE] EXTERNAL TABLE <tbl name> (<attr list>)
+LOCATION ('pxf://<HA nameservice>/<path to file or directory>?Profile=profile[&<additional options>=<value>]')
+FORMAT '[TEXT | CSV | CUSTOM]' (<formatting properties>);
+```
+
+The opposite is true when a highly available HDFS cluster is reverted to a single namenode configuration. In that case, any table definition that has the nameservice specified should use the <NN host>:<NN rest port> syntax.
+
+## Using a Record Key with Key-Value File Formats<a id="recordkeyinkey-valuefileformats"></a>
+
+For sequence file and other file formats that store rows in a key-value format, the key value can be accessed through HAWQ by using the saved keyword '`recordkey`' as a field name.
+
+The field type must correspond to the key type, much as the other fields must match the HDFS data.
+
+WritableResolver supports read and write of recordkey, which can be of the following Writable Hadoop types:
+
+- BooleanWritable
+- ByteWritable
+- DoubleWritable
+- FloatWritable
+- IntWritable
+- LongWritable
+- Text
+
+If the `recordkey` field is not defined, the key is ignored in read, and a default value (segment id as LongWritable) is written in write.
+
+### Example<a id="example1"></a>
+
+A data schema `Babies.class` contains three fields: (name text, birthday text, weight float). An external table must include these three fields, and can either include or ignore the recordkey.
+
+``` sql
+-- writable table with recordkey
+CREATE WRITABLE EXTERNAL TABLE babies_registry (recordkey int, name text, birthday text, weight float)
+ LOCATION ('pxf://namenode_host:51200/babies_1940s'
+ '?ACCESSOR=org.apache.hawq.pxf.plugins.hdfs.SequenceFileAccessor'
+ '&RESOLVER=org.apache.hawq.pxf.plugins.hdfs.WritableResolver'
+ '&DATA-SCHEMA=Babies')
+ FORMAT 'CUSTOM' (formatter='pxfwritable_export');
+INSERT INTO babies_registry VALUES (123456, "James Paul McCartney", "June 18, 1942", 3.800);
+
+-- writable table without recordkey
+CREATE WRITABLE EXTERNAL TABLE babies_registry2 (name text, birthday text, weight float)
+ LOCATION ('pxf://namenode_host:51200/babies_1940s'
+ '?ACCESSOR=org.apache.hawq.pxf.plugins.SequenceFileAccessor'
+ '&RESOLVER=org.apache.hawq.pxf.plugins.WritableResolver'
+ '&DATA-SCHEMA=Babies')
+ FORMAT 'CUSTOM' (formatter='pxfwritable_export');
+
+-- this record's key will have some default value
+INSERT INTO babies_registry VALUES ("Richard Starkey", "July 7, 1940", 4.0);
+```
+
+The same goes for reading data from an existing file with a key-value format, e.g. a Sequence file.
+
+``` sql
+-- readable table with recordkey
+CREATE EXTERNAL TABLE babies_1940 (recordkey int, name text, birthday text, weight float)
+ LOCATION ('pxf://namenode_host:51200/babies_1940s'
+ '?FRAGMENTER=org.apache.hawq.pxf.plugins.hdfs.HdfsDataFragmenter'
+ '&ACCESSOR=org.apache.hawq.pxf.plugins.hdfs.SequenceFileAccessor'
+ '&RESOLVER=org.apache.hawq.pxf.plugins.hdfs.WritableResolver'
+ '&DATA-SCHEMA=Babies')
+ FORMAT 'CUSTOM' (formatter='pxfwritable_import');
+-- retrieve each record's key
+SELECT * FROM babies_1940;
+
+-- readable table without recordkey
+CREATE EXTERNAL TABLE babies_1940_2 (name text, birthday text, weight float)
+ LOCATION ('pxf://namenode_host:51200/babies_1940s'
+ '?FRAGMENTER=org.apache.hawq.pxf.plugins.hdfs.HdfsDataFragmenter'
+ '&ACCESSOR=org.apache.hawq.pxf.plugins.hdfs.SequenceFileAccessor'
+ '&RESOLVER=org.apache.hawq.pxf.plugins.hdfs.WritableResolver'
+ '&DATA-SCHEMA=Babies')
+ FORMAT 'CUSTOM' (formatter='pxfwritable_import');
+-- ignores the records' key
+SELECT * FROM babies_1940_2;
+```
+
+## Working with Avro Files<a id="topic_oy3_qwm_ss"></a>
+
+Avro files combine their data with a schema, and can support complex data types such as arrays, maps, records, enumerations, and fixed types. When you create a PXF external table to represent Avro data, map top-level fields in the schema that use a primitive data type to HAWQ columns of the same type. Map top-level fields that use a complex data type to a TEXT column in the external table. The PXF Avro profile automatically separates components of a complex type by inserting delimiters in the text column. You can then use functions or application code to further process components of the complex data.
+
+The following table summarizes external table mapping rules for Avro data.
+
+<caption><span class="tablecap">Table 2. Avro Data Type Mapping</span></caption>
+
+<a id="topic_oy3_qwm_ss__table_j4s_h1n_ss"></a>
+
+| Avro Data Type | PXF Type |
+|-------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Primitive type (int, double, float, long, string, bytes, boolean) | Corresponding HAWQ data type. See [Data Types](../reference/HAWQDataTypes.html). |
+| Complex type: Array, Map, Record, or Enum | TEXT, with default delimiters inserted between collection items, mapped key-value pairs, and record data. |
+| Complex type: Fixed | BYTEA |
+| Union | Follows the above conventions for primitive or complex data types, depending on the union. Null values are supported in Unions. |
+
+For complex types, the PXF Avro profile inserts default delimiters between collection items and values. You can use non-default delimiter characters by including the `COLLECTION_DELIM`, `MAPKEY_DELIM`, and/or `RECORDKEY_DELIM` optional parameters on the Avro profile. See [Additional PXF Options](#additionaloptions__table_skq_kpz_4p) for a description of the parameters.
+
+### Example<a id="topic_tr3_dpg_ts"></a>
+
+The following example uses the Avro schema shown in [Sample Avro Schema](#topic_tr3_dpg_ts__section_m2p_ztg_ts) and the associated data file shown in [Sample Avro Data (JSON)](#topic_tr3_dpg_ts__section_spk_15g_ts).
+
+#### Sample Avro Schema<a id="topic_tr3_dpg_ts__section_m2p_ztg_ts"></a>
+
+``` json
+{
+ "type" : "record",
+ "name" : "example_schema",
+ "namespace" : "com.example",
+ "fields" : [ {
+ "name" : "id",
+ "type" : "long",
+ "doc" : "Id of the user account"
+ }, {
+ "name" : "username",
+ "type" : "string",
+ "doc" : "Name of the user account"
+ }, {
+ "name" : "followers",
+ "type" : {"type": "array", "items": "string"},
+ "doc" : "Users followers"
+ }, {
+ "name": "rank",
+ "type": ["null", "int"],
+ "default": null
+ }, {
+ "name": "fmap",
+ "type": {"type": "map", "values": "long"}
+ }, {
+ "name": "address",
+ "type": {
+ "type": "record",
+ "name": "addressRecord",
+ "fields": [
+ {"name":"number", "type":"int"},
+ {"name":"street", "type":"string"},
+ {"name":"city", "type":"string"}]
+ }
+ }, {
+ "name": "relationship",
+ "type": {
+ "type": "enum",
+ "name": "relationshipEnum",
+ "symbols": ["MARRIED","LOVE","FRIEND","COLLEAGUE","STRANGER","ENEMY"]
+ }
+ }, {
+ "name" : "md5",
+ "type": {
+ "type" : "fixed",
+ "name" : "md5Fixed",
+ "size" : 4
+ }
+ } ],
+ "doc:" : "A basic schema for storing messages"
+}
+```
+
+#### Sample Avro Data (JSON)<a id="topic_tr3_dpg_ts__section_spk_15g_ts"></a>
+
+``` pre
+{"id":1, "username":"john","followers":["kate", "santosh"], "rank":null, "relationship": "FRIEND", "fmap": {"kate":10,"santosh":4},
+"address":{"street":"renaissance drive", "number":1,"city":"san jose"}, "md5":\u3F00\u007A\u0073\u0074}
+
+{"id":2, "username":"jim","followers":["john", "pam"], "rank":3, "relationship": "COLLEAGUE", "fmap": {"john":3,"pam":3},
+"address":{"street":"deer creek", "number":9,"city":"palo alto"}, "md5":\u0010\u0021\u0003\u0004}
+```
+
+To map this Avro file to an external table, the top-level primitive fields ("id" of type long and "username" of type string) are mapped to their equivalent HAWQ types (bigint and text). The remaining complex fields are mapped to text columns:
+
+``` sql
+gpadmin=# CREATE EXTERNAL TABLE avro_complex
+ (id bigint,
+ username text,
+ followers text,
+ rank int,
+ fmap text,
+ address text,
+ relationship text,
+ md5 bytea)
+LOCATION ('pxf://namehost:51200/tmp/avro_complex?PROFILE=Avro')
+FORMAT 'CUSTOM' (FORMATTER='pxfwritable_import');
+```
+
+The above command uses default delimiters for separating components of the complex types. This command is equivalent to the one above, but it explicitly sets the delimiters using the Avro profile parameters:
+
+``` sql
+gpadmin=# CREATE EXTERNAL TABLE avro_complex
+ (id bigint,
+ username text,
+ followers text,
+ rank int,
+ fmap text,
+ address text,
+ relationship text,
+ md5 bytea)
+LOCATION ('pxf://localhost:51200/tmp/avro_complex?PROFILE=Avro&COLLECTION_DELIM=,&MAPKEY_DELIM=:&RECORDKEY_DELIM=:')
+FORMAT 'CUSTOM' (FORMATTER='pxfwritable_import');
+```
+
+A simple query of the external table shows the components of the complex type data separated with delimiters:
+
+``` sql
+gpadmin=# select * from avro_complex;
+```
+
+``` pre
+id | username | followers | rank | fmap | address | relationship | md5
+1| john | [kate,santosh] | | {kate:10,santosh:4} | {number:1,street:renaissance drive,city:san jose} | FRIEND | ?zst
+2| jim | [john,pam] | 3 | {john:3,pam:3} | {number:9,street:deer creek,city:palo alto} | COLLEAGUE | \020!\003\004
+```
+
+You can process the delimited components in the text columns as necessary for your application. For example, the following command uses the `string_to_array` function to convert entries in the "followers" field to a text array column in a new view. The view is then queried to filter rows based on whether a particular follower appears in the array:
+
+``` sql
+gpadmin=# create view followers_view as
+ select username, address, string_to_array(substring(followers from 2 for (char_length(followers) - 2)), ',')::text[]
+ as followers
+ from avro_complex;
+
+gpadmin=# select username, address
+from followers_view
+where john = ANY(followers);
+```
+
+``` pre
+username | address
+jim | {number:9,street:deer creek,city:palo alto}
+```
diff --git a/pxf/HivePXF.html.md.erb b/pxf/HivePXF.html.md.erb
new file mode 100644
index 0000000..521b7f1
--- /dev/null
+++ b/pxf/HivePXF.html.md.erb
@@ -0,0 +1,401 @@
+---
+title: Accessing Hive Data
+---
+
+This topic describes how to access Hive data using PXF. You have several options for querying data stored in Hive. You can create external tables in PXF and then query those tables, or you can easily query Hive tables by using HAWQ and PXF's integration with HCatalog. HAWQ accesses Hive table metadata stored in HCatalog.
+
+## Prerequisites<a id="installingthepxfhiveplugin"></a>
+
+Check the following before using PXF to access Hive:
+
+- The PXF HDFS plug-in is installed on all cluster nodes.
+- The PXF Hive plug-in is installed on all cluster nodes.
+- The Hive JAR files and conf directory are installed on all cluster nodes.
+- Test PXF on HDFS before connecting to Hive or HBase.
+- You are running the Hive Metastore service on a machine in your cluster.
+- You have set the `hive.metastore.uris` property in the `hive-site.xml` on the NameNode.
+
+## Hive Command Line<a id="hivecommandline"></a>
+
+To start the Hive command line and work directly on a Hive table:
+
+``` shell
+$ ${HIVE_HOME}/bin/hive
+```
+
+Here is an example of how to create and load data into a sample Hive table from an existing file.
+
+``` sql
+Hive> CREATE TABLE test (name string, type string, supplier_key int, full_price double) row format delimited fields terminated by ',';
+Hive> LOAD DATA local inpath '/local/path/data.txt' into table test;
+```
+
+## Using PXF Tables to Query Hive<a id="topic_p2s_lvl_25"></a>
+
+Hive tables are defined in a specific way in PXF, regardless of the underlying file storage format. The PXF Hive plug-ins automatically detect source tables in the following formats:
+
+- Text based
+- SequenceFile
+- RCFile
+- ORCFile
+- Parquet
+- Avro
+
+The source table can also be a combination of these types. The PXF Hive plug-in uses this information to query the data in runtime.
+
+- **[Syntax Example](../pxf/HivePXF.html#syntax2)**
+
+- **[Hive Complex Types](../pxf/HivePXF.html#topic_b4v_g3n_25)**
+
+### Syntax Example<a id="syntax2"></a>
+
+The following PXF table definition is valid for any Hive file storage type.
+
+``` sql
+CREATE [READABLE|WRITABLE] EXTERNAL TABLE table_name
+ ( column_name data_type [, ...] | LIKE other_table )
+LOCATION ('pxf://namenode[:port]/hive-table-name?<pxf parameters>[&custom-option=value...]')FORMAT 'CUSTOM' (formatter='pxfwritable_import')
+```
+
+where `<pxf parameters>` is:
+
+``` pre
+ FRAGMENTER=fragmenter_class&ACCESSOR=accessor_class&RESOLVER=resolver_class]
+ | PROFILE=profile-name
+```
+
+**Note:** The port is the connection port for the PXF service. If the port is omitted, PXF assumes that High Availability (HA) is enabled and connects to the HA name service port, 51200 by default. The HA name service port can be changed by setting thepxf\_service\_port configuration parameter.
+
+PXF has three built-in profiles for Hive tables:
+
+- Hive
+- HiveRC
+- HiveText
+
+The Hive profile works with any Hive storage type. Use HiveRC and HiveText to query RC and Text formats respectively. The HiveRC and HiveText profiles are faster than the generic Hive profile. When using the HiveRC and HiveText profiles, you must specify a DELIMITER option in the LOCATION clause. See [Using Profiles to Read and Write Data](ReadWritePXF.html#readingandwritingdatawithpxf) for more information on profiles.
+
+The following example creates a readable external table for a Hive table named `/user/eddie/test` using the PXF Hive profile:
+
+``` sql
+CREATE EXTERNAL TABLE hivetest(id int, newid int)
+LOCATION ('pxf://namenode:51200/user/eddie/test?PROFILE=Hive')
+FORMAT 'custom' (formatter='pxfwritable_import');
+```
+
+### Hive Complex Types<a id="topic_b4v_g3n_25"></a>
+
+PXF tables support Hive data types that are not primitive types. The supported Hive complex data types are array, struct, map, and union. This Hive `CREATE TABLE` statement, for example, creates a table with each of these complex data types:
+
+``` sql
+CREATE TABLE sales_collections (
+ item STRING,
+ price FLOAT,
+ properties ARRAY<STRING>,
+ hash MAP<STRING,FLOAT>,
+ delivery_address STRUCT<street:STRING, city:STRING, state:STRING, zip:INT>,
+ others UNIONTYPE<FLOAT, BOOLEAN, STRING>
+)
+ROW FORMAT DELIMITED FIELDS
+TERMINATED BY '\001' COLLECTION ITEMS TERMINATED BY '\002' MAP KEYS TERMINATED BY '\003' LINES TERMINATED BY '\n' STORED AS TEXTFILE;
+LOAD DATA LOCAL INPATH '/local/path/<some data file>' INTO TABLE sales_collection;
+```
+
+You can use HAWQ functions or application code to extract the components of the complex data columns as needed.
+
+## Using PXF and HCatalog to Query Hive<a id="hcatalog"></a>
+
+Hive tables can be queried directly through HCatalog integration with HAWQ and PXF, regardless of the underlying file storage format.
+
+Previously, in order to query Hive tables using HAWQ and PXF, you needed to create an external table in PXF that described the target table's Hive metadata. Since HAWQ is now integrated with HCatalog, HAWQ can use metadata stored in HCatalog instead of external tables created for PXF. HCatalog is built on top of the Hive metastore and incorporates Hive's DDL. This provides several advantages:
+
+- You do not need to know the table schema of your Hive tables
+- You do not need to manually enter information about Hive table location or format
+- If Hive table metadata changes, HCatalog provides updated metadata. This is in contrast to the use of static external PXF tables to define Hive table metadata for HAWQ.
+
+The following diagram depicts how HAWQ integrates with HCatalog to query Hive tables:
+
+<img src="../images/hawq_hcatalog.png" id="hcatalog__image_ukw_h2v_c5" class="image" width="672" />
+
+1. HAWQ retrieves table metadata from HCatalog using PXF.
+2. HAWQ creates in-memory catalog tables from the retrieved metadata. If a table is referenced multiple times in a transaction, HAWQ uses its in-memory metadata to reduce external calls to HCatalog.
+3. PXF queries Hive using table metadata that is stored in the HAWQ in-memory catalog tables. Table metadata is dropped at the end of the transaction.
+
+- **[Usage](../pxf/HivePXF.html#topic_j1l_y55_c5)**
+
+- **[Limitations](../pxf/HivePXF.html#topic_r5k_pst_25)**
+
+### Usage<a id="topic_j1l_y55_c5"></a>
+
+To enable HCatalog query integration in HAWQ, perform the following steps:
+
+1. Make sure your deployment meets the requirements listed in [Prerequisites](#installingthepxfhiveplugin).
+2. If necessary, set the `pxf_service_address` global configuration property to a hostname or IP address and port where you have installed the PXF Hive plug-in. By default, the value is set to `localhost:51200`.
+
+ ``` sql
+ SET pxf_service_address TO "hivenode:51200"
+ ```
+
+3. To query a Hive table with HCatalog integration, simply query HCatalog directly. The query syntax is:
+
+ ``` sql
+ SELECT * FROM hcatalog.hive-db-name.hive-table-name;
+ ```
+
+ For example:
+
+ ``` sql
+ SELECT * FROM hcatalog.default.sales;
+ ```
+
+4. To obtain a description of a Hive table with HCatalog integration, you can use the `psql` client interface.
+ - Within HAWQ, use either the `\d hcatalog.hive-db-name.hive-table-name` or `\d+ hcatalog.hive-db-name.hive-table-name` commands to describe a single table. For example, from the `psql` client interface:
+
+ ``` shell
+ $ psql -d postgres
+ postgres=# \d hcatalog.default.test
+
+ PXF Hive Table "default.test"
+ Column | Type
+ --------------+--------
+ name | text
+ type | text
+ supplier_key | int4
+ full_price | float8
+ ```
+ - Use `\d hcatalog.hive-db-name.*` to describe the whole database schema. For example:
+
+ ``` shell
+ postgres=# \d hcatalog.default.*
+
+ PXF Hive Table "default.test"
+ Column | Type
+ --------------+--------
+ type | text
+ name | text
+ supplier_key | int4
+ full_price | float8
+
+ PXF Hive Table "default.testabc"
+ Column | Type
+ --------+------
+ type | text
+ name | text
+ ```
+ - Use `\d hcatalog.*.*` to describe the whole schema:
+
+ ``` shell
+ postgres=# \d hcatalog.*.*
+
+ PXF Hive Table "default.test"
+ Column | Type
+ --------------+--------
+ type | text
+ name | text
+ supplier_key | int4
+ full_price | float8
+
+ PXF Hive Table "default.testabc"
+ Column | Type
+ --------+------
+ type | text
+ name | text
+
+ PXF Hive Table "userdb.test"
+ Column | Type
+ ----------+------
+ address | text
+ username | text
+
+ ```
+
+ **Note:** When using `\d` or `\d+` commands in the `psql` HAWQ client, `hcatalog` will not be listed as a database. If you use other `psql` compatible clients, `hcatalog` will be listed as a database with a size value of `-1` since `hcatalog` is not a real database in HAWQ.
+
+5. Alternatively, you can use the **pxf\_get\_item\_fields** user-defined function (UDF) to obtain Hive table descriptions from other client interfaces or third-party applications. The UDF takes a PXF profile and a table pattern string as its input parameters.
+
+ **Note:** Currently the only supported input profile is `'Hive'`.
+
+ For example, the following statement returns a description of a specific table. The description includes path, itemname (table), fieldname, and fieldtype.
+
+ ``` sql
+ # select * from pxf_get_item_fields('Hive','default.test');
+ ```
+
+ ``` pre
+ path | itemname | fieldname | fieldtype
+ ---------+----------+--------------+-----------
+ default | test | name | text
+ default | test | type | text
+ default | test | supplier_key | int4
+ default | test | full_price | float8
+ (4 rows)
+ ```
+
+ The following statement returns table descriptions from the default database.
+
+ ``` sql
+ postgres=# select * from pxf_get_item_fields('Hive','default.*');
+ ```
+
+ ``` pre
+ path | itemname | fieldname | fieldtype
+ ---------+----------+--------------+-----------
+ default | test | name | text
+ default | test | type | text
+ default | test | supplier_key | int4
+ default | test | full_price | float8
+ default | testabc | name | text
+ default | testabc | type | text
+ (6 rows)
+ ```
+
+ The following statement returns a description of the entire schema.
+
+ ``` sql
+ # select * from pxf_get_item_fields('Hive', '*.*');
+ ```
+
+ ``` pre
+ path | itemname | fieldname | fieldtype
+ ---------+----------+--------------+-----------
+ default | test | name | text
+ default | test | type | text
+ default | test | supplier_key | int4
+ default | test | full_price | float8
+ default | testabc | name | text
+ default | testabc | type | text
+ userdb | test | username | text
+ userdb | test | address | text
+ (8 rows)
+ ```
+
+### Limitations<a id="topic_r5k_pst_25"></a>
+
+HCatalog integration has the following limitations:
+
+- HCatalog integration queries and describe commands do not support complex types; only primitive types are supported. Use PXF external tables to query complex types in Hive instead. (See [Hive Complex Types](#topic_b4v_g3n_25) for example.)
+- Even for primitive types, HCatalog metadata descriptions produced by `\d` and` \d+` are converted to HAWQ types. For example, the Hive type `tinyint` is converted to HAWQ type `int2`. (See [Data Types](../reference/HAWQDataTypes.html) for a list of data types in HAWQ.)
+- HAWQ reserves the database name "hcatalog" for system use. You cannot connect to or alter the system "hcatalog" database.
+
+## Partition Filtering<a id="partitionfiltering"></a>
+
+The PXF Hive plug-in uses the Hive partitioning feature and directory structure. This enables partition exclusion on HDFS files that contain the Hive table. To use the partition filtering feature to reduce network traffic and I/O, run a PXF query using a WHERE clause that refers to a specific partition in the partitioned Hive table.
+
+To take advantage of PXF partition filtering push-down, name the partition fields in the external table. These names must be the same as the names stored in the Hive table. Otherwise, PXF ignores Partition filtering and the filtering is performed on the HAWQ side, impacting performance.
+
+**Note:** The Hive plug-in only filters on partition columns, not on other table attributes.
+
+### Example<a id="example2"></a>
+
+Create a Hive table `sales_part` with two partition columns, `delivery_state` and `delivery_city:`
+
+``` sql
+CREATE TABLE sales_part (name STRING, type STRING, supplier_key INT, price DOUBLE)
+PARTITIONED BY (delivery_state STRING, delivery_city STRING)
+ROW FORMAT DELIMITED FIELDS TERMINATED BY ',';
+```
+
+Load data into this Hive table and add some partitions:
+
+``` sql
+LOAD DATA LOCAL INPATH '/local/path/data1.txt' INTO TABLE sales_part PARTITION(delivery_state = 'CALIFORNIA', delivery_city = 'San Francisco');
+LOAD DATA LOCAL INPATH '/local/path/data2.txt' INTO TABLE sales_part PARTITION(delivery_state = 'CALIFORNIA', delivery_city = 'Sacramento');
+LOAD DATA LOCAL INPATH '/local/path/data3.txt' INTO TABLE sales_part PARTITION(delivery_state = 'NEVADA' , delivery_city = 'Reno');
+LOAD DATA LOCAL INPATH '/local/path/data4.txt' INTO TABLE sales_part PARTITION(delivery_state = 'NEVADA' , delivery_city = 'Las Vegas');
+```
+
+The Hive storage directory should appears as follows:
+
+``` pre
+/hive/warehouse/sales_part/delivery_state=CALIFORNIA/delivery_city=’San Francisco’/data1.txt
+/hive/warehouse/sales_part/delivery_state=CALIFORNIA/delivery_city=Sacramento/data2.txt
+/hive/warehouse/sales_part/delivery_state=NEVADA/delivery_city=Reno/data3.txt
+/hive/warehouse/sales_part/delivery_state=NEVADA/delivery_city=’Las Vegas’/data4.txt
+```
+
+To define a PXF table that will read this Hive table and take advantage of partition filter push-down, define the fields corresponding to the Hive partition fields at the end of the attribute list. In HiveQL, a `select *` statement on a partitioned table shows the partition fields at the end of the record.
+
+``` sql
+CREATE EXTERNAL TABLE pxf_sales_part(
+ item_name TEXT,
+ item_type TEXT,
+ supplier_key INTEGER,
+ item_price DOUBLE PRECISION,
+ delivery_state TEXT,
+ delivery_city TEXT
+)
+LOCATION ('pxf://namenode_host:51200/sales_part?Profile=Hive')
+FORMAT 'custom' (FORMATTER='pxfwritable_import');
+
+SELECT * FROM pxf_sales_part;
+```
+
+### Example<a id="example3"></a>
+
+In the following example, the HAWQ query filters the `delivery_city` partition `Sacramento`. The filter on `item_name` is not pushed down, since it is not a partition column. It is performed on the HAWQ side after all the data on `Sacramento` is transferred for processing.
+
+``` sql
+SELECT * FROM pxf_sales_part WHERE delivery_city = 'Sacramento' AND item_name = 'shirt';
+```
+
+### Example<a id="example4"></a>
+
+The following HAWQ query reads all the data under `delivery_state` partition `CALIFORNIA`, regardless of the city.
+
+``` sql
+SELECT * FROM pxf_sales_part WHERE delivery_state = 'CALIFORNIA';
+```
+
+## Using PXF with Hive Default Partitions<a id="topic_fdm_zrh_1s"></a>
+
+This topic describes a difference in query results between Hive and PXF queries when Hive tables use a default partition. When dynamic partitioning is enabled in Hive, a partitioned table may store data in a default partition. Hive creates a default partition when the value of a partitioning column does not match the defined type of the column (for example, when a NULL value is used for any partitioning column). In Hive, any query that includes a filter on a partition column *excludes* any data that is stored in the table's default partition.
+
+Similar to Hive, PXF represents a table's partitioning columns as columns that are appended to the end of the table. However, PXF translates any column value in a default partition to a NULL value. This means that a HAWQ query that includes an IS NULL filter on a partitioning column can return different results than the same Hive query.
+
+### Example<a id="topic_g4r_4wh_1s"></a>
+
+Consider a Hive partitioned table that is created with the statement:
+
+``` sql
+CREATE TABLE sales (order_id bigint, order_amount float) PARTITIONED BY (date date);
+```
+
+The table is loaded with five rows that contain the following data:
+
+``` pre
+1 1.0 1900-01-01
+2 2.2 1994-04-14
+3 3.3 2011-03-31
+4 4.5 NULL
+5 5.0 2013-12-06
+```
+
+In this case, the insertion of row 4 creates a Hive default partition, because the partition column "date" contains a null value.
+
+In Hive, any query that filters on the partition column omits data in the default partition. For example the following query returns no rows:
+
+``` sql
+hive> select * from sales where date is null;
+```
+
+However, if you map this table as a PXF external table in HAWQ, all default partition values are translated into actual NULL values. In HAWQ, executing the same query against the PXF table returns row 4 as the result, because the filter matches the NULL value.
+
+Keep this behavior in mind when executing IS NULL queries on Hive partitioned tables.
+
+## Accessing Hive Tables in Parquet Format<a id="topic_dbb_nz3_ts"></a>
+
+The PXF Hive profile supports both non-partitioned and partitioned Hive tables that use the Parquet storage format in HDFS. Simply map the table columns using equivalent HAWQ data types. For example, in Hive if a table is created using:
+
+``` sql
+hive> create table hive_parquet_table (fname string, lname string, custid int, acctbalance double) stored as
+ parquet;
+```
+
+Then you would define the HAWQ external table using:
+
+``` sql
+create external table pxf_parquet_table (fname text, lname text, custid int, acctbalance double precision)
+ location ('pxf://localhost:51200/hive_parquet_table?profile=Hive')
+ format 'custom' (formatter='pxfwritable_import');
+```
+
+
diff --git a/pxf/InstallPXFPlugins.html.md.erb b/pxf/InstallPXFPlugins.html.md.erb
new file mode 100644
index 0000000..199a30e
--- /dev/null
+++ b/pxf/InstallPXFPlugins.html.md.erb
@@ -0,0 +1,120 @@
+---
+title: Installing PXF Plug-ins
+---
+
+This topic describes how to install the built-in PXF service plug-ins that are required to connect PXF to HDFS, Hive, and HBase.
+
+**Note:** The PXF plug-ins require that you run Tomcat on the host machine. Tomcat reserves ports 8005, 8080, and 8009. If you have configured Oozie JXM reporting on a host that will run a PXF plug-in, make sure that the reporting service uses a port other than 8005. This helps to prevent port conflict errors from occurring when you start the PXF service.
+
+## PXF Installation and Log File Directories<a id="directories_and_logs"></a>
+
+Installing PXF plug-ins, regardless of method, creates directories and log files on each node receiving the plug-in installation:
+
+| Directory | Description |
+|--------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `/usr/lib/pxf` | PXF library location |
+| `/etc/pxf/conf` | PXF configuration directory. This directory contains the `pxf-public.classpath` and `pxf-private.classpath` configuration files. See [Setting up the Java Classpath](ConfigurePXF.html#settingupthejavaclasspath). |
+| `/var/pxf/pxf-service` | PXF service instance location |
+| `/var/log/pxf` | This directory includes `pxf-service.log` and all Tomcat-related logs including `catalina.out`. Logs are owned by user:group `pxf`:`pxf`. Other users have read access. |
+| `/var/run/pxf/catalina.pid` | PXF Tomcat container PID location |
+
+
+## Installing PXF Plug-ins Using Ambari<a id="install_pxf_plug_ambari"></a>
+
+If you are using Ambari to install and manage your HAWQ cluster, you do *not* need to follow the manual installation steps in this topic. Installing using the Ambari web interface installs all of the necessary PXF plug-in components.
+
+## Installing PXF Plug-ins from the Command Line<a id="install_pxf_plug_cmdline"></a>
+
+Each PXF service plug-in resides in its own RPM. You may have built these RPMs in the Apache HAWQ open source project repository (see [PXF Build Instructions](https://github.com/apache/incubator-hawq/blob/master/pxf/README.md)), or these RPMs may have been included in a commercial product download package.
+
+RPMs for PXF plug-ins must be installed on *each* node in your cluster.
+
+### Installing Prerequisite Packages<a id="topic_qsz_wq5_25"></a>
+All PXF plug-ins require that both the Tomcat and PXF service packages be installed on each node in your cluster.
+
+Install Tomcat:
+
+```shell
+$ sudo rpm -i apache-tomcat-x.x.x.noarch.rpm
+```
+
+where *x.x.x* corresponds to the version of Tomcat required by PXF. The appropriate version of Tomcat is included in the PXF RPM bundle.
+
+Install the PXF service:
+
+```shell
+$ sudo rpm -i pxf-service-n.n.n-x.noarch.rpm
+```
+
+where *n.n.n-x* corresponds to the PXF version and build number you wish to install.
+
+Installing the PXF service package:
+
+ - creates a `/usr/lib/pxf-n.n.n` directory, adding a softlink from `/usr/lib/pxf` to this directory
+ - copies the PXF service JAR file `pxf-service-n.n.n.jar` to `/usr/lib/pxf-n.n.n/`
+ - creates a softlink `pxf-service.jar` in `/usr/lib/pxf-n.n.n/`
+ - sets up the PXF service configuration files
+ - starts the PXF service
+
+
+### Installing the PXF HDFS Plug-in<a id="topic_qsz_wq5_25"></a>
+
+To install PXF support for HDFS, perform the following steps on *each* node in your cluster:
+
+1. Install the Tomcat and PXF service packages as described in the previous section.
+
+2. Install the PXF HDFS plug-in:
+
+ ```shell
+ $ sudo rpm -i pxf-hdfs-n.n.n-x.noarch.rpm
+ ```
+
+ The install copies the HDFS JAR file `pxf-hdfs-n.n.n.jar` to `/usr/lib/pxf-n.n.n/` and creates a softlink `pxf-hdfs.jar` in that directory.
+
+### Installing the PXF Hive Plug-in<a id="topic_s1y_1v5_25"></a>
+
+To install PXF support for Hive, perform the following steps on *each* node in your cluster:
+
+1. Install the Tomcat, PXF service, and PXF HDFS RPMs as previously described.
+
+2. Install the PXF Hive plug-in:
+
+ ```shell
+ $ sudo rpm -i pxf-hive-n.n.n-x.noarch.rpm
+ ```
+
+ The install copies the Hive JAR file `pxf-hive-n.n.n.jar` to `/usr/lib/pxf-n.n.n` and creates a softlink `pxf-hive.jar` in that directory.
+
+### Installing the PXF HBase Plug-in<a id="topic_std_cv5_25"></a>
+
+To install PXF support for HBase, perform the following steps on *each* node in your cluster.
+
+1. Install the Tomcat, PXF Service, and PXF HDFS RPMs as previously described.
+
+2. Install the PXF HBase plug-in:
+
+ ```shell
+ $ sudo rpm -i pxf-hbase-n.n.n-x.noarch.rpm
+ ```
+
+ The install copies the HBase JAR file `pxf-hbase-n.n.n.jar` to `/usr/lib/pxf-n.n.n` and creates a softlink `pxf-hbase.jar` in that directory.
+
+3. Add the PXF HBase plug-in JAR file to the HBase `CLASSPATH` by updating the `HBASE_CLASSPATH` environment variable setting in the HBase environment file `/etc/hbase/conf/hbase-env.sh`:
+
+ ```shell
+ export HBASE_CLASSPATH=${HBASE_CLASSPATH}:/usr/lib/pxf/pxf-hbase.jar
+ ```
+
+4. Restart the HBase service after making this update to HBase configuration.
+
+ If you are on the HBase Master node:
+
+ ```shell
+ $ su -l hbase -c "/usr/hdp/current/hbase-master/bin/hbase-daemon.sh restart master; sleep 25"
+ ```
+
+ If you are on an HBase Region Server node:
+
+ ```shell
+ $ su -l hbase -c "/usr/hdp/current/hbase-regionserver/bin/hbase-daemon.sh restart regionserver"
+ ```
diff --git a/pxf/PXFExternalTableandAPIReference.html.md.erb b/pxf/PXFExternalTableandAPIReference.html.md.erb
new file mode 100644
index 0000000..157622f
--- /dev/null
+++ b/pxf/PXFExternalTableandAPIReference.html.md.erb
@@ -0,0 +1,1338 @@
+---
+title: PXF External Tables and API
+---
+
+You can use the PXF API to create your own connectors to access any other type of parallel data store or processing engine.
+
+The PXF Java API lets you extend PXF functionality and add new services and formats without changing HAWQ. The API includes three classes that are extended to allow HAWQ to access an external data source: Fragmenter, Accessor, and Resolver.
+
+The Fragmenter produces a list of data fragments that can be read in parallel from the data source. The Accessor produces a list of records from a single fragment, and the Resolver both deserializes and serializes records.
+
+Together, the Fragmenter, Accessor, and Resolver classes implement a connector. PXF includes plug-ins for tables in HDFS, HBase, and Hive.
+
+## Creating an External Table<a id="creatinganexternaltable"></a>
+
+The syntax for a readable `EXTERNAL TABLE` that uses the PXF protocol is as follows:
+
+``` sql
+CREATE [READABLE|WRITABLE] EXTERNAL TABLE table_name
+ ( column_name data_type [, ...] | LIKE other_table )
+LOCATION('pxf://host[:port]/path-to-data<pxf parameters>[&custom-option=value...]')
+FORMAT 'custom' (formatter='pxfwritable_import|pxfwritable_export');
+```
+
+ where *<pxf parameters>* is:
+
+``` pre
+ ?FRAGMENTER=fragmenter_class&ACCESSOR=accessor_class&RESOLVER=resolver_class]
+ | ?PROFILE=profile-name
+```
+<caption><span class="tablecap">Table 1. Parameter values and description</span></caption>
+
+<a id="creatinganexternaltable__table_pfy_htz_4p"></a>
+
+| Parameter | Value and description |
+|-------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| host | The current host of the PXF service. |
+| port | Connection port for the PXF service. If the port is omitted, PXF assumes that High Availability (HA) is enabled and connects to the HA name service port, 51200 by default. The HA name service port can be changed by setting the `pxf_service_port` configuration parameter. |
+| *path\_to\_data* | A directory, file name, wildcard pattern, table name, etc. |
+| FRAGMENTER | The plug-in (Java class) to use for fragmenting data. Used for READABLE external tables only. |
+| ACCESSOR | The plug-in (Java class) to use for accessing the data. Used for READABLE and WRITABLE tables. |
+| RESOLVER | The plug-in (Java class) to use for serializing and deserializing the data. Used for READABLE and WRITABLE tables. |
+| *custom-option*=*value* | Additional values to pass to the plug-in class. The parameters are passed at runtime to the plug-ins indicated above. The plug-ins can lookup custom options with `org.apache.hawq.pxf.api.utilities.InputData`. |
+
+**Note:** When creating PXF external tables, you cannot use the `HEADER` option in your `FORMAT` specification.
+
+For more information about this example, see [About the Java Class Services and Formats](#aboutthejavaclassservicesandformats).
+
+## About the Java Class Services and Formats<a id="aboutthejavaclassservicesandformats"></a>
+
+The `LOCATION` string in a PXF `CREATE EXTERNAL TABLE` statement is a URI that specifies the host and port of an external data source and the path to the data in the external data source. The query portion of the URI, introduced by the question mark (?), must include the required parameters `FRAGMENTER` (readable tables only), `ACCESSOR`, and `RESOLVER`, which specify Java class names that extend the base PXF API plug-in classes. Alternatively, the required parameters can be replaced with a `PROFILE` parameter with the name of a profile defined in the `/etc/conf/pxf-profiles.xml` that defines the required classes.
+
+The parameters in the PXF URI are passed from HAWQ as headers to the PXF Java service. You can pass custom information to user-implemented PXF plug-ins by adding optional parameters to the LOCATION string.
+
+The Java PXF service retrieves the source data from the external data source and converts it to a HAWQ-readable table format.
+
+The Accessor, Resolver, and Fragmenter Java classes extend the `org.apache.hawq.pxf.api.utilities.Plugin` class:
+
+``` java
+package org.apache.hawq.pxf.api.utilities;
+/**
+ * Base class for all plug-in types (Accessor, Resolver, Fragmenter, ...).
+ * Manages the meta data.
+ */
+public class Plugin {
+ protected InputData inputData;
+ /**
+ * Constructs a plug-in.
+ *
+ * @param input the input data
+ */
+ public Plugin(InputData input) {
+ this.inputData = input;
+ }
+ /**
+ * Checks if the plug-in is thread safe or not, based on inputData.
+ *
+ * @return true if plug-in is thread safe
+ */
+ public boolean isThreadSafe() {
+ return true;
+ }
+}
+```
+
+The parameters in the `LOCATION` string are available to the plug-ins through methods in the `org.apache.hawq.pxf.api.utilities.InputData` class. Custom parameters added to the location string can be looked up with the `getUserProperty()` method.
+
+``` java
+/**
+ * Common configuration available to all PXF plug-ins. Represents input data
+ * coming from client applications, such as HAWQ.
+ */
+public class InputData {
+
+ /**
+ * Constructs an InputData from a copy.
+ * Used to create from an extending class.
+ *
+ * @param copy the input data to copy
+ */
+ public InputData(InputData copy);
+
+ /**
+ * Returns value of a user defined property.
+ *
+ * @param userProp the lookup user property
+ * @return property value as a String
+ */
+ public String getUserProperty(String userProp);
+
+ /**
+ * Sets the byte serialization of a fragment meta data
+ * @param location start, len, and location of the fragment
+ */
+ public void setFragmentMetadata(byte[] location);
+
+ /** Returns the byte serialization of a data fragment */
+ public byte[] getFragmentMetadata();
+
+ /**
+ * Gets any custom user data that may have been passed from the
+ * fragmenter. Will mostly be used by the accessor or resolver.
+ */
+ public byte[] getFragmentUserData();
+
+ /**
+ * Sets any custom user data that needs to be shared across plug-ins.
+ * Will mostly be set by the fragmenter.
+ */
+ public void setFragmentUserData(byte[] userData);
+
+ /** Returns the number of segments in GP. */
+ public int getTotalSegments();
+
+ /** Returns the current segment ID. */
+ public int getSegmentId();
+
+ /** Returns true if there is a filter string to parse. */
+ public boolean hasFilter();
+
+ /** Returns the filter string, <tt>null</tt> if #hasFilter is <tt>false</tt> */
+ public String getFilterString();
+
+ /** Returns tuple description. */
+ public ArrayList<ColumnDescriptor> getTupleDescription();
+
+ /** Returns the number of columns in tuple description. */
+ public int getColumns();
+
+ /** Returns column index from tuple description. */
+ public ColumnDescriptor getColumn(int index);
+
+ /**
+ * Returns the column descriptor of the recordkey column. If the recordkey
+ * column was not specified by the user in the create table statement will
+ * return null.
+ */
+ public ColumnDescriptor getRecordkeyColumn();
+
+ /** Returns the data source of the required resource (i.e a file path or a table name). */
+ public String getDataSource();
+
+ /** Sets the data source for the required resource */
+ public void setDataSource(String dataSource);
+
+ /** Returns the ClassName for the java class that was defined as Accessor */
+ public String getAccessor();
+
+ /** Returns the ClassName for the java class that was defined as Resolver */
+ public String getResolver();
+
+ /**
+ * Returns the ClassName for the java class that was defined as Fragmenter
+ * or null if no fragmenter was defined
+ */
+ public String getFragmenter();
+
+ /**
+ * Returns the contents of pxf_remote_service_login set in Hawq.
+ * Should the user set it to an empty string this function will return null.
+ *
+ * @return remote login details if set, null otherwise
+ */
+ public String getLogin();
+
+ /**
+ * Returns the contents of pxf_remote_service_secret set in Hawq.
+ * Should the user set it to an empty string this function will return null.
+ *
+ * @return remote password if set, null otherwise
+ */
+ public String getSecret();
+
+ /**
+ * Returns true if the request is thread safe. Default true. Should be set
+ * by a user to false if the request contains non thread-safe plug-ins or
+ * components, such as BZip2 codec.
+ */
+ public boolean isThreadSafe();
+
+ /**
+ * Returns a data fragment index. plan to deprecate it in favor of using
+ * getFragmentMetadata().
+ */
+ public int getDataFragment();
+}
+```
+
+- **[Fragmenter](../pxf/PXFExternalTableandAPIReference.html#fragmenter)**
+
+- **[Accessor](../pxf/PXFExternalTableandAPIReference.html#accessor)**
+
+- **[Resolver](../pxf/PXFExternalTableandAPIReference.html#resolver)**
+
+### Fragmenter<a id="fragmenter"></a>
+
+**Note:** The Fragmenter Plugin reads data into HAWQ readable external tables. The Fragmenter Plugin cannot write data out of HAWQ into writable external tables.
+
+The Fragmenter is responsible for passing datasource metadata back to HAWQ. It also returns a list of data fragments to the Accessor or Resolver. Each data fragment describes some part of the requested data set. It contains the datasource name, such as the file or table name, including the hostname where it is located. For example, if the source is a HDFS file, the Fragmenter returns a list of data fragments containing a HDFS file block. Each fragment includes the location of the block. If the source data is an HBase table, the Fragmenter returns information about table regions, including their locations.
+
+The `ANALYZE` command now retrieves advanced statistics for PXF readable tables by estimating the number of tuples in a table, creating a sample table from the external table, and running advanced statistics queries on the sample table in the same way statistics are collected for native HAWQ tables.
+
+The configuration parameter `pxf_enable_stat_collection` controls collection of advanced statistics. If `pxf_enable_stat_collection` is set to false, no analysis is performed on PXF tables. An additional parameter, `pxf_stat_max_fragments`, controls the number of fragments sampled to build a sample table. By default `pxf_stat_max_fragments` is set to 100, which means that even if there are more than 100 fragments, only this number of fragments will be used in `ANALYZE` to sample the data. Increasing this number will result in better sampling, but can also impact performance.
+
+When a PXF table is analyzed and `pxf_enable_stat_collection` is set to off, or an error occurs because the table is not defined correctly, the PXF service is down, or `getFragmentsStats` is not implemented, a warning message is shown and no statistics are gathered for that table. If `ANALYZE` is running over all tables in the database, the next table will be processed – a failure processing one table does not stop the command.
+
+For a detailed explanation about HAWQ statistical data gathering, see `ANALYZE` in the SQL Commands Reference.
+
+**Note:**
+
+- Depending on external table size, the time required to complete an ANALYZE operation can be lengthy. The boolean parameter `pxf_enable_stat_collection` enables statistics collection for PXF. The default value is `on`. Turning this parameter off (disabling PXF statistics collection) can help decrease the time needed for the ANALYZE operation.
+- You can also use *pxf\_stat\_max\_fragments* to limit the number of fragments to be sampled by decreasing it from the default (100). However, if the number is too low, the sample might not be uniform and the statistics might be skewed.
+- You can also implement getFragmentsStats to return an error. This will cause ANALYZE on a table with this Fragmenter to fail immediately, and default statistics values will be used for that table.
+
+The following table lists the Fragmenter plug-in implementations included with the PXF API.
+
+<a id="fragmenter__table_cgs_svp_3s"></a>
+
+<table>
+<caption><span class="tablecap">Table 2. Fragmenter base classes </span></caption>
+<colgroup>
+<col width="50%" />
+<col width="50%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th><p><code class="ph codeph">Fragmenter class</code></p></th>
+<th><p><code class="ph codeph">Description</code></p></th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>org.apache.hawq.pxf.plugins.hdfs.HdfsDataFragmenter</td>
+<td>Fragmenter for Hdfs files</td>
+</tr>
+<tr class="even">
+<td>org.apache.hawq.pxf.plugins.hbase.HBaseAtomicDataAccessor</td>
+<td>Fragmenter for HBase tables</td>
+</tr>
+<tr class="odd">
+<td>org.apache.hawq.pxf.plugins.hive.HiveDataFragmenter</td>
+<td>Fragmenter for Hive tables </td>
+</tr>
+<tr class="even">
+<td>org.apache.hawq.pxf.plugins.hdfs.HiveInputFormatFragmenter</td>
+<td>Fragmenter for Hive tables with RC or text files </td>
+</tr>
+</tbody>
+</table>
+
+A Fragmenter class extends `org.apache.hawq.pxf.api.Fragmenter`:
+
+#### org.apache.hawq.pxf.api.Fragmenter<a id="com.pivotal.pxf.api.fragmenter"></a>
+
+``` java
+package org.apache.hawq.pxf.api;
+/**
+ * Abstract class that defines the splitting of a data resource into fragments
+ * that can be processed in parallel.
+ */
+public abstract class Fragmenter extends Plugin {
+ protected List<Fragment> fragments;
+
+ public Fragmenter(InputData metaData) {
+ super(metaData);
+ fragments = new LinkedList<Fragment>();
+ }
+
+ /**
+ * Gets the fragments of a given path (source name and location of each
+ * fragment). Used to get fragments of data that could be read in parallel
+ * from the different segments.
+ */
+ public abstract List<Fragment> getFragments() throws Exception;
+
+ /**
+ * Default implementation of statistics for fragments. The default is:
+ * <ul>
+ * <li>number of fragments - as gathered by {@link #getFragments()}</li>
+ * <li>first fragment size - 64MB</li>
+ * <li>total size - number of fragments times first fragment size</li>
+ * </ul>
+ * Each fragmenter implementation can override this method to better match
+ * its fragments stats.
+ *
+ * @return default statistics
+ * @throws Exception if statistics cannot be gathered
+ */
+ public FragmentsStats getFragmentsStats() throws Exception {
+ List<Fragment> fragments = getFragments();
+ long fragmentsNumber = fragments.size();
+ return new FragmentsStats(fragmentsNumber,
+ FragmentsStats.DEFAULT_FRAGMENT_SIZE, fragmentsNumber
+ * FragmentsStats.DEFAULT_FRAGMENT_SIZE);
+ }
+}
+
+```
+
+`getFragments()` returns a string in JSON format of the retrieved fragment. For example, if the input path is a HDFS directory, the source name for each fragment should include the file name including the path for the fragment.
+
+#### Class Description<a id="classdescription"></a>
+
+The Fragmenter.getFragments() method returns a List<Fragment>;:
+
+``` java
+package org.apache.hawq.pxf.api;
+/*
+ * Fragment holds a data fragment' information.
+ * Fragmenter.getFragments() returns a list of fragments.
+ */
+public class Fragment
+{
+ private String sourceName; // File path+name, table name, etc.
+ private int index; // Fragment index (incremented per sourceName)
+ private String[] replicas; // Fragment replicas (1 or more)
+ private byte[] metadata; // Fragment metadata information (starting point + length, region location, etc.)
+ private byte[] userData; // ThirdParty data added to a fragment. Ignored if null
+ ...
+}
+```
+
+#### org.apache.hawq.pxf.api.FragmentsStats<a id="topic_fzd_tlv_c5"></a>
+
+The `Fragmenter.getFragmentsStats()` method returns a `FragmentsStats`:
+
+``` java
+package org.apache.hawq.pxf.api;
+/**
+ * FragmentsStats holds statistics for a given path.
+ */
+public class FragmentsStats {
+
+ // number of fragments
+ private long fragmentsNumber;
+ // first fragment size
+ private SizeAndUnit firstFragmentSize;
+ // total fragments size
+ private SizeAndUnit totalSize;
+
+ /**
+ * Enum to represent unit (Bytes/KB/MB/GB/TB)
+ */
+ public enum SizeUnit {
+ /**
+ * Byte
+ */
+ B,
+ /**
+ * KB
+ */
+ KB,
+ /**
+ * MB
+ */
+ MB,
+ /**
+ * GB
+ */
+ GB,
+ /**
+ * TB
+ */
+ TB;
+ };
+
+ /**
+ * Container for size and unit
+ */
+ public class SizeAndUnit {
+ long size;
+ SizeUnit unit;
+ ...
+
+```
+
+`getFragmentsStats()` returns a string in JSON format of statistics for the data source. For example, if the input path is a HDFS directory of 3 files, each one of 1 block, the output will be the number of fragments (3), the size of the first file, and the size of all files in that directory.
+
+### Accessor<a id="accessor"></a>
+
+The Accessor retrieves specific fragments and passes records back to the Resolver. For example, the HDFS plug-ins create a `org.apache.hadoop.mapred.FileInputFormat` and a `org.apache.hadoop.mapred.RecordReader` for an HDFS file and sends this to the Resolver. In the case of HBase or Hive files, the Accessor returns single rows from an HBase or Hive table. PXF 1.x or higher contains the following Accessor implementations:
+
+<a id="accessor__table_ewm_ttz_4p"></a>
+
+<table>
+<caption><span class="tablecap">Table 3. Accessor base classes </span></caption>
+<colgroup>
+<col width="50%" />
+<col width="50%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th><p><code class="ph codeph">Accessor class</code></p></th>
+<th><p><code class="ph codeph">Description</code></p></th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>org.apache.hawq.pxf.plugins.hdfs.HdfsAtomicDataAccessor</td>
+<td>Base class for accessing datasources which cannot be split. These will be accessed by a single HAWQ segment</td>
+</tr>
+<tr class="even">
+<td>org.apache.hawq.pxf.plugins.hdfs.QuotedLineBreakAccessor</td>
+<td>Accessor for TEXT files that have records with embedded linebreaks</td>
+</tr>
+<tr class="odd">
+<td>org.apache.hawq.pxf.plugins.hdfs.HdfsSplittableDataAccessor</td>
+<td><p>Base class for accessing HDFS files using <code class="ph codeph">RecordReaders</code></p></td>
+</tr>
+<tr class="even">
+<td>org.apache.hawq.pxf.plugins.hdfs.LineBreakAccessor</td>
+<td>Accessor for TEXT files (replaced the deprecated <code class="ph codeph">TextFileAccessor</code>, <code class="ph codeph">LineReaderAccessor</code>)</td>
+</tr>
+<tr class="odd">
+<td>org.apache.hawq.pxf.plugins.hdfs.AvroFileAccessor</td>
+<td>Accessor for Avro files</td>
+</tr>
+<tr class="even">
+<td>org.apache.hawq.pxf.plugins.hdfs.SequenceFileAccessor</td>
+<td>Accessor for Sequence files</td>
+</tr>
+<tr class="odd">
+<td>org.apache.hawq.pxf.plugins.hbase.HBaseAccessor </td>
+<td>Accessor for HBase tables </td>
+</tr>
+<tr class="even">
+<td>org.apache.hawq.pxf.plugins.hive.HiveAccessor</td>
+<td>Accessor for Hive tables </td>
+</tr>
+<tr class="odd">
+<td>org.apache.hawq.pxf.plugins.hive.HiveLineBreakAccessor</td>
+<td>Accessor for Hive tables with text files</td>
+</tr>
+<tr class="even">
+<td>org.apache.hawq.pxf.plugins.hive.HiveRCFileAccessor</td>
+<td>Accessor for Hive tables with RC files</td>
+</tr>
+</tbody>
+</table>
+
+The class must extend the `org.apache.hawq.pxf.Plugin` class, and implement one or both interfaces:
+
+- `org.apache.hawq.pxf.api.ReadAccessor`
+- `org.apache.hawq.pxf.api.WriteAccessor`
+
+``` java
+package org.apache.hawq.pxf.api;
+/*
+ * Internal interface that defines the access to data on the source
+ * data store (e.g, a file on HDFS, a region of an HBase table, etc).
+ * All classes that implement actual access to such data sources must
+ * respect this interface
+ */
+public interface ReadAccessor {
+ boolean openForRead() throws Exception;
+ OneRow readNextObject() throws Exception;
+ void closeForRead() throws Exception;
+}
+```
+
+``` java
+package org.apache.hawq.pxf.api;
+/*
+ * An interface for writing data into a data store
+ * (e.g, a sequence file on HDFS).
+ * All classes that implement actual access to such data sources must
+ * respect this interface
+ */
+public interface WriteAccessor {
+ boolean openForWrite() throws Exception;
+ OneRow writeNextObject(OneRow onerow) throws Exception;
+ void closeForWrite() throws Exception;
+}
+```
+
+The Accessor calls `openForRead()` to read existing data. After reading the data, it calls `closeForRead()`. `readNextObject()` returns one of the following:
+
+- a single record, encapsulated in a OneRow object
+- null if it reaches `EOF`
+
+The Accessor calls `openForWrite()` to write data out. After writing the data, it writes a `OneRow` object with `writeNextObject()`, and when done calls `closeForWrite()`. `OneRow` represents a key-value item.
+
+#### org.apache.hawq.pxf.api.OneRow<a id="com.pivotal.pxf.api.onerow"></a>
+
+``` java
+package org.apache.hawq.pxf.api;
+/*
+ * Represents one row in the external system data store. Supports
+ * the general case where one row contains both a record and a
+ * separate key like in the HDFS key/value model for MapReduce
+ * (Example: HDFS sequence file)
+ */
+public class OneRow {
+ /*
+ * Default constructor
+ */
+ public OneRow();
+
+ /*
+ * Constructor sets key and data
+ */
+ public OneRow(Object inKey, Object inData);
+
+ /*
+ * Setter for key
+ */
+ public void setKey(Object inKey);
+
+ /*
+ * Setter for data
+ */
+ public void setData(Object inData);
+
+ /*
+ * Accessor for key
+ */
+ public Object getKey();
+
+ /*
+ * Accessor for data
+ */
+ public Object getData();
+
+ /*
+ * Show content
+ */
+ public String toString();
+}
+```
+
+### Resolver<a id="resolver"></a>
+
+The Resolver deserializes records in the `OneRow` format and serializes them to a list of `OneField` objects. PXF converts a `OneField` object to a HAWQ-readable `GPDBWritable` format. PXF 1.x or higher contains the following implementations:
+
+<a id="resolver__table_nbd_d5z_4p"></a>
+
+<table>
+<caption><span class="tablecap">Table 4. Resolver base classes</span></caption>
+<colgroup>
+<col width="50%" />
+<col width="50%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th><p><code class="ph codeph">Resolver class</code></p></th>
+<th><p><code class="ph codeph">Description</code></p></th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td><p><code class="ph codeph">org.apache.hawq.pxf.plugins.hdfs.StringPassResolver</code></p></td>
+<td><p><code class="ph codeph">StringPassResolver</code> replaced the deprecated <code class="ph codeph">TextResolver</code>. It passes whole records (composed of any data types) as strings without parsing them</p></td>
+</tr>
+<tr class="even">
+<td><p><code class="ph codeph">org.apache.hawq.pxf.plugins.hdfs.WritableResolver</code></p></td>
+<td><p>Resolver for custom Hadoop Writable implementations. Custom class can be specified with the schema in DATA-SCHEMA. Supports the following types:</p>
+<pre class="pre codeblock"><code>DataType.BOOLEAN
+DataType.INTEGER
+DataType.BIGINT
+DataType.REAL
+DataType.FLOAT8
+DataType.VARCHAR
+DataType.BYTEA</code></pre></td>
+</tr>
+<tr class="odd">
+<td><p><code class="ph codeph">org.apache.hawq.pxf.plugins.hdfs.AvroResolver</code></p></td>
+<td><p>Supports the same field objects as <code class="ph codeph">WritableResolver</code>. </p></td>
+</tr>
+<tr class="even">
+<td><p><code class="ph codeph">org.apache.hawq.pxf.plugins.hbase.HBaseResolver</code></p></td>
+<td><p>Supports the same field objects as <code class="ph codeph">WritableResolver</code> and also supports the following:</p>
+<pre class="pre codeblock"><code>DataType.SMALLINT
+DataType.NUMERIC
+DataType.TEXT
+DataType.BPCHAR
+DataType.TIMESTAMP</code></pre></td>
+</tr>
+<tr class="odd">
+<td><p><code class="ph codeph">org.apache.hawq.pxf.plugins.hive.HiveResolver</code></p></td>
+<td><p>Supports the same field objects as <code class="ph codeph">WritableResolver</code> and also supports the following:</p>
+<pre class="pre codeblock"><code>DataType.SMALLINT
+DataType.TEXT
+DataType.TIMESTAMP</code></pre></td>
+</tr>
+<tr class="even">
+<td><p><code class="ph codeph">org.apache.hawq.pxf.plugins.hive.HiveStringPassResolver</code></p></td>
+<td>Specialized <code class="ph codeph">HiveResolver</code> for a Hive table stored as Text files. Should be used together with <code class="ph codeph">HiveInputFormatFragmenter</code>/<code class="ph codeph">HiveLineBreakAccessor</code>.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">org.apache.hawq.pxf.plugins.hive.HiveColumnarSerdeResolver</code></td>
+<td>Specialized <code class="ph codeph">HiveResolver</code> for a Hive table stored as RC file. Should be used together with <code class="ph codeph">HiveInputFormatFragmenter</code>/<code class="ph codeph">HiveRCFileAccessor</code>.</td>
+</tr>
+</tbody>
+</table>
+
+The class needs to extend the `org.apache.hawq.pxf.resolvers.Plugin class `, and implement one or both interfaces:
+
+- `org.apache.hawq.pxf.api.ReadResolver`
+- `org.apache.hawq.pxf.api.WriteResolver`
+
+``` java
+package org.apache.hawq.pxf.api;
+/*
+ * Interface that defines the deserialization of one record brought from
+ * the data Accessor. Every implementation of a deserialization method
+ * (e.g, Writable, Avro, ...) must implement this interface.
+ */
+public interface ReadResolver {
+ public List<OneField> getFields(OneRow row) throws Exception;
+}
+```
+
+``` java
+package org.apache.hawq.pxf.api;
+/*
+* Interface that defines the serialization of data read from the DB
+* into a OneRow object.
+* Every implementation of a serialization method
+* (e.g, Writable, Avro, ...) must implement this interface.
+*/
+public interface WriteResolver {
+ public OneRow setFields(List<OneField> record) throws Exception;
+}
+```
+
+**Note:**
+
+- getFields should return a List<OneField>, each OneField representing a single field.
+- `setFields `should return a single `OneRow `object, given a List<OneField>.
+
+#### org.apache.hawq.pxf.api.OneField<a id="com.pivotal.pxf.api.onefield"></a>
+
+``` java
+package org.apache.hawq.pxf.api;
+/*
+ * Defines one field on a deserialized record.
+ * 'type' is in OID values recognized by GPDBWritable
+ * 'val' is the actual field value
+ */
+public class OneField {
+ public OneField() {}
+ public OneField(int type, Object val) {
+ this.type = type;
+ this.val = val;
+ }
+
+ public int type;
+ public Object val;
+}
+```
+
+The value of `type` should follow the org.apache.hawq.pxf.api.io.DataType `enums`. `val` is the appropriate Java class. Supported types are as follows:
+
+<a id="com.pivotal.pxf.api.onefield__table_f4x_35z_4p"></a>
+
+<table>
+<caption><span class="tablecap">Table 5. Resolver supported types</span></caption>
+<colgroup>
+<col width="50%" />
+<col width="50%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th><p>DataType recognized OID</p></th>
+<th><p>Field value</p></th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td><p><code class="ph codeph">DataType.SMALLINT</code></p></td>
+<td><p><code class="ph codeph">Short</code></p></td>
+</tr>
+<tr class="even">
+<td><p><code class="ph codeph">DataType.INTEGER</code></p></td>
+<td><p><code class="ph codeph">Integer</code></p></td>
+</tr>
+<tr class="odd">
+<td><p><code class="ph codeph">DataType.BIGINT</code></p></td>
+<td><p><code class="ph codeph">Long</code></p></td>
+</tr>
+<tr class="even">
+<td><p><code class="ph codeph">DataType.REAL</code></p></td>
+<td><p><code class="ph codeph">Float</code></p></td>
+</tr>
+<tr class="odd">
+<td><p><code class="ph codeph">DataType.FLOAT8</code></p></td>
+<td><p><code class="ph codeph">Double</code></p></td>
+</tr>
+<tr class="even">
+<td><p><code class="ph codeph">DataType.NUMERIC</code></p></td>
+<td><p><code class="ph codeph">String ("651687465135468432168421")</code></p></td>
+</tr>
+<tr class="odd">
+<td><p><code class="ph codeph">DataType.BOOLEAN</code></p></td>
+<td><p><code class="ph codeph">Boolean</code></p></td>
+</tr>
+<tr class="even">
+<td><p><code class="ph codeph">DataType.VARCHAR</code></p></td>
+<td><p><code class="ph codeph">String</code></p></td>
+</tr>
+<tr class="odd">
+<td><p><code class="ph codeph">DataType.BPCHAR</code></p></td>
+<td><p><code class="ph codeph">String</code></p></td>
+</tr>
+<tr class="even">
+<td><p><code class="ph codeph">DataType.TEXT</code></p></td>
+<td><p><code class="ph codeph">String</code></p></td>
+</tr>
+<tr class="odd">
+<td><p><code class="ph codeph">DataType.BYTEA</code></p></td>
+<td><p><code class="ph codeph">byte []</code></p></td>
+</tr>
+<tr class="even">
+<td><p><code class="ph codeph">DataType.TIMESTAMP</code></p></td>
+<td><p><code class="ph codeph">Timestamp</code></p></td>
+</tr>
+<tr class="odd">
+<td><p><code class="ph codeph">DataType.Date</code></p></td>
+<td><p><code class="ph codeph">Date</code></p></td>
+</tr>
+</tbody>
+</table>
+
+### Analyzer<a id="analyzer"></a>
+
+The Analyzer has been deprecated. A new function in the Fragmenter API (Fragmenter.getFragmentsStats) is used to gather initial statistics for the data source, and provides PXF statistical data for the HAWQ query optimizer. For a detailed explanation about HAWQ statistical data gathering, see `ANALYZE` in the SQL Command Reference.
+
+Using the Analyzer API will result in an error message. Use the Fragmenter and getFragmentsStats to gather advanced statistics.
+
+## About Custom Profiles<a id="aboutcustomprofiles"></a>
+
+Administrators can add new profiles or edit the built-in profiles in `/etc/conf/pxf-profiles.xml` file. See [Using Profiles to Read and Write Data](ReadWritePXF.html#readingandwritingdatawithpxf) for information on how to add custom profiles.
+
+## About Query Filter Push-Down<a id="aboutqueryfilterpush-down"></a>
+
+If a query includes a number of WHERE clause filters, HAWQ may push all or some queries to PXF. If pushed to PXF, the Accessor can use the filtering information when accessing the data source to fetch tuples. These filters only return records that pass filter evaluation conditions. This reduces data processing and reduces network traffic from the SQL engine.
+
+This topic includes the following information:
+
+- Filter Availability and Ordering
+- Creating a Filter Builder class
+- Filter Operations
+- Sample Implementation
+- Using Filters
+
+### Filter Availability and Ordering<a id="filteravailabilityandordering"></a>
+
+PXF allows push-down filtering if the following rules are met:
+
+- Uses only single expressions or a group of AND'ed expressions - no OR'ed expressions.
+- Uses only expressions of supported data types and operators.
+
+FilterParser scans the pushed down filter list and uses the user's build() implementation to build the filter.
+
+- For simple expressions (e.g, a >= 5), FilterParser places column objects on the left of the expression and constants on the right.
+- For compound expressions (e.g <expression> AND <expression>) it handles three cases in the build() function:
+ 1. Simple Expression: <Column Index> <Operation> <Constant>
+ 2. Compound Expression: <Filter Object> AND <Filter Object>
+ 3. Compound Expression: <List of Filter Objects> AND <Filter Object>
+
+### Creating a Filter Builder Class<a id="creatingafilterbuilderclass"></a>
+
+To check if a filter queried PXF, call the `InputData hasFilter()` function:
+
+``` java
+/*
+ * Returns true if there is a filter string to parse
+ */
+public boolean hasFilter()
+{
+ return filterStringValid;
+}
+```
+
+If `hasFilter()` returns `false`, there is no filter information. If it returns `true`, PXF parses the serialized filter string into a meaningful filter object to use later. To do so, create a filter builder class that implements the `FilterParser.FilterBuilder ` interface:
+
+``` java
+package org.apache.hawq.pxf.api;
+/*
+ * Interface a user of FilterParser should implement
+ * This is used to let the user build filter expressions in the manner she
+ * sees fit
+ *
+ * When an operator is parsed, this function is called to let the user decide
+ * what to do with its operands.
+ */
+interface FilterBuilder {
+ public Object build(Operation operation, Object left, Object right) throws Exception;
+}
+```
+
+While PXF parses the serialized filter string from the incoming HAWQ query, it calls the `build() interface` function. PXF calls this function for each condition or filter pushed down to PXF. Implementing this function returns some Filter object or representation that the Fragmenter, Accessor, or Resolver uses in runtime to filter out records. The `build()` function accepts an Operation as input, and left and right operands.
+
+### Filter Operations<a id="filteroperations"></a>
+
+``` java
+/*
+ * Operations supported by the parser
+ */
+public enum Operation
+{
+ HDOP_LT, //less than
+ HDOP_GT, //greater than
+ HDOP_LE, //less than or equal
+ HDOP_GE, //greater than or equal
+ HDOP_EQ, //equal
+ HDOP_NE, //not equal
+ HDOP_AND //AND'ed conditions
+};
+```
+
+#### Filter Operands<a id="filteroperands"></a>
+
+There are three types of operands:
+
+- Column Index
+- Constant
+- Filter Object
+
+#### Column Index<a id="columnindex"></a>
+
+``` java
+/*
+ * Represents a column index
+ */
+public class ColumnIndex
+{
+ public ColumnIndex(int idx);
+
+ public int index();
+}
+```
+
+#### Constant<a id="constant"></a>
+
+``` java
+/*
+ * The class represents a constant object (String, Long, ...)
+ */
+public class Constant
+{
+ public Constant(Object obj);
+
+ public Object constant();
+}
+```
+
+#### Filter Object<a id="filterobject"></a>
+
+Filter Objects can be internal, such as those you define; or external, those that the remote system uses. For example, for HBase, you define the HBase `Filter` class (`org.apache.hadoop.hbase.filter.Filter`), while for Hive, you use an internal default representation created by the PXF framework, called `BasicFilter`. You can decide the filter object to use, including writing a new one. `BasicFilter` is the most common:
+
+``` java
+/*
+ * Basic filter provided for cases where the target storage system does not provide its own filter
+ * For example: Hbase storage provides its own filter but for a Writable based record in a SequenceFile
+ * there is no filter provided and so we need to have a default
+ */
+static public class BasicFilter
+{
+ /*
+ * C'tor
+ */
+ public BasicFilter(Operation inOper, ColumnIndex inColumn, Constant inConstant);
+
+ /*
+ * Returns oper field
+ */
+ public Operation getOperation();
+
+ /*
+ * Returns column field
+ */
+ public ColumnIndex getColumn();
+
+ /*
+ * Returns constant field
+ */
+ public Constant getConstant();
+}
+```
+
+### Sample Implementation<a id="sampleimplementation"></a>
+
+Let's look at the following sample implementation of the filter builder class and its `build()` function that handles all 3 cases. Let's assume that BasicFilter was used to hold our filter operations.
+
+``` java
+import java.util.LinkedList;
+import java.util.List;
+
+import org.apache.hawq.pxf.api.FilterParser;
+import org.apache.hawq.pxf.api.utilities.InputData;
+
+public class MyDemoFilterBuilder implements FilterParser.FilterBuilder
+{
+ private InputData inputData;
+
+ public MyDemoFilterBuilder(InputData input)
+ {
+ inputData = input;
+ }
+
+ /*
+ * Translates a filterString into a FilterParser.BasicFilter or a list of such filters
+ */
+ public Object getFilterObject(String filterString) throws Exception
+ {
+ FilterParser parser = new FilterParser(this);
+ Object result = parser.parse(filterString);
+
+ if (!(result instanceof FilterParser.BasicFilter) && !(result instanceof List))
+ throw new Exception("String " + filterString + " resolved to no filter");
+
+ return result;
+ }
+
+ public Object build(FilterParser.Operation opId,
+ Object leftOperand,
+ Object rightOperand) throws Exception
+ {
+ if (leftOperand instanceof FilterParser.BasicFilter)
+ {
+ //sanity check
+ if (opId != FilterParser.Operation.HDOP_AND || !(rightOperand instanceof FilterParser.BasicFilter))
+ throw new Exception("Only AND is allowed between compound expressions");
+
+ //case 3
+ if (leftOperand instanceof List)
+ return handleCompoundOperations((List<FilterParser.BasicFilter>)leftOperand, (FilterParser.BasicFilter)rightOperand);
+ //case 2
+ else
+ return handleCompoundOperations((FilterParser.BasicFilter)leftOperand, (FilterParser.BasicFilter)rightOperand);
+ }
+
+ //sanity check
+ if (!(rightOperand instanceof FilterParser.Constant))
+ throw new Exception("expressions of column-op-column are not supported");
+
+ //case 1 (assume column is on the left)
+ return handleSimpleOperations(opId, (FilterParser.ColumnIndex)leftOperand, (FilterParser.Constant)rightOperand);
+ }
+
+ private FilterParser.BasicFilter handleSimpleOperations(FilterParser.Operation opId,
+ FilterParser.ColumnIndex column,
+ FilterParser.Constant constant)
+ {
+ return new FilterParser.BasicFilter(opId, column, constant);
+ }
+
+ private List handleCompoundOperations(List<FilterParser.BasicFilter> left,
+ FilterParser.BasicFilter right)
+ {
+ left.add(right);
+ return left;
+ }
+
+ private List handleCompoundOperations(FilterParser.BasicFilter left,
+ FilterParser.BasicFilter right)
+ {
+ List<FilterParser.BasicFilter> result = new LinkedList<FilterParser.BasicFilter>();
+
+ result.add(left);
+ result.add(right);
+ return result;
+ }
+}
+```
+
+Here is an example of creating a filter-builder class to implement the Filter interface, implement the `build()` function, and generate the Filter object. To do this, use either the Accessor, Resolver, or both to call the `getFilterObject` function:
+
+``` java
+if (inputData.hasFilter())
+{
+ String filterStr = inputData.filterString();
+ MyDemoFilterBuilder demobuilder = new MyDemoFilterBuilder(inputData);
+ Object filter = demobuilder.getFilterObject(filterStr);
+ ...
+}
+```
+
+### Using Filters<a id="usingfilters"></a>
+
+Once you have built the Filter object(s), you can use them to read data and filter out records that do not meet the filter conditions:
+
+1. Check whether you have a single or multiple filters.
+2. Evaluate each filter and iterate over each filter in the list. Disqualify the record if filter conditions fail.
+
+``` java
+if (filter instanceof List)
+{
+ for (Object f : (List)filter)
+ <evaluate f>; //may want to break if evaluation results in negative answer for any filter.
+}
+else
+{
+ <evaluate filter>;
+}
+```
+
+Example of evaluating a single filter:
+
+``` java
+//Get our BasicFilter Object
+FilterParser.BasicFilter bFilter = (FilterParser.BasicFilter)filter;
+
+
+//Get operation and operator values
+FilterParser.Operation op = bFilter.getOperation();
+int colIdx = bFilter.getColumn().index();
+String val = bFilter.getConstant().constant().toString();
+
+//Get more info about the column if desired
+ColumnDescriptor col = input.getColumn(colIdx);
+String colName = filterColumn.columnName();
+
+//Now evaluate it against the actual column value in the record...
+```
+
+## Examples<a id="reference"></a>
+
+This section contains the following information:
+
+- [External Table Examples](#externaltableexamples)
+- [Plug-in Examples](#pluginexamples)
+
+- **[External Table Examples](../pxf/PXFExternalTableandAPIReference.html#externaltableexamples)**
+
+- **[Plug-in Examples](../pxf/PXFExternalTableandAPIReference.html#pluginexamples)**
+
+### External Table Examples<a id="externaltableexamples"></a>
+
+#### Example 1<a id="example1"></a>
+
+Shows an external table that can analyze all `Sequencefiles` that are populated `Writable` serialized records and exist inside the hdfs directory `sales/2012/01`. `SaleItem.class` is a Java class that implements the `Writable` interface and describes a Java record that includes three class members.
+
+**Note:** In this example, the class member names do not necessarily match the database attribute names, but the types match. `SaleItem.class` must exist in the classpath of every DataNode and NameNode.
+
+``` sql
+CREATE EXTERNAL TABLE jan_2012_sales (id int, total int, comments varchar)
+LOCATION ('pxf://10.76.72.26:51200/sales/2012/01/*.seq'
+ '?FRAGMENTER=org.apache.hawq.pxf.plugins.hdfs.HdfsDataFragmenter'
+ '&ACCESSOR=org.apache.hawq.pxf.plugins.hdfs.SequenceFileAccessor'
+ '&RESOLVER=org.apache.hawq.pxf.plugins.hdfs.WritableResolver'
+ '&DATA-SCHEMA=SaleItem')
+FORMAT 'custom' (formatter='pxfwritable_import');
+```
+
+#### Example 2<a id="example2"></a>
+
+Example 2 shows an external table that can analyze an HBase table called `sales`. It has 10 column families `(cf1 – cf10)` and many qualifier names in each family. This example focuses on the `rowkey`, the qualifier `saleid` inside column family `cf1`, and the qualifier `comments` inside column family `cf8` and uses direct mapping:
+
+``` sql
+CREATE EXTERNAL TABLE hbase_sales
+ (hbaserowkey text, "cf1:saleid" int, "cf8:comments" varchar)
+LOCATION ('pxf://10.76.72.26:51200/sales?PROFILE=HBase')
+FORMAT 'custom' (formatter='pxfwritable_import');
+```
+
+#### Example 3<a id="example3"></a>
+
+This example uses indirect mapping. Note how the attribute name changes and how they correspond to the HBase lookup table. Executing `SELECT FROM my_hbase_sales`, the attribute names automatically convert to their HBase correspondents.
+
+``` sql
+CREATE EXTERNAL TABLE my_hbase_sales (hbaserowkey text, id int, cmts varchar)
+LOCATION
+('pxf://10.76.72.26:51200/sales?PROFILE=HBase')
+FORMAT 'custom' (formatter='pxfwritable_import');
+```
+
+#### Example 4<a id="example4"></a>
+
+Shows an example for a writable table of compressed data.
+
+``` sql
+CREATE WRITABLE EXTERNAL TABLE sales_aggregated_2012
+ (id int, total int, comments varchar)
+LOCATION ('pxf://10.76.72.26:51200/sales/2012/aggregated'
+ '?PROFILE=HdfsTextSimple'
+ '&COMPRESSION_CODEC=org.apache.hadoop.io.compress.BZip2Codec')
+FORMAT 'TEXT';
+```
+
+#### Example 5<a id="example5"></a>
+
+Shows an example for a writable table into a sequence file, using a schema file. For writable tables, the formatter is `pxfwritable_export`.
+
+``` sql
+CREATE WRITABLE EXTERNAL TABLE sales_max_2012
+ (id int, total int, comments varchar)
+LOCATION ('pxf://10.76.72.26:51200/sales/2012/max'
+ '?FRAGMENTER=org.apache.hawq.pxf.plugins.hdfs.HdfsDataFragmenter'
+ '&ACCESSOR=org.apache.hawq.pxf.plugins.hdfs.SequenceFileAccessor'
+ '&RESOLVER=org.apache.hawq.pxf.plugins.hdfs.WritableResolver'
+ '&DATA-SCHEMA=SaleItem')
+FORMAT 'custom' (formatter='pxfwritable_export');
+```
+
+### Plug-in Examples<a id="pluginexamples"></a>
+
+This section contains sample dummy implementations of all three plug-ins. It also contains a usage example.
+
+#### Dummy Fragmenter<a id="dummyfragmenter"></a>
+
+``` java
+import org.apache.hawq.pxf.api.Fragmenter;
+import org.apache.hawq.pxf.api.Fragment;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import java.util.List;
+
+/*
+ * Class that defines the splitting of a data resource into fragments that can
+ * be processed in parallel
+ * getFragments() returns the fragments information of a given path (source name and location of each fragment).
+ * Used to get fragments of data that could be read in parallel from the different segments.
+ * Dummy implementation, for documentation
+ */
+public class DummyFragmenter extends Fragmenter {
+ public DummyFragmenter(InputData metaData) {
+ super(metaData);
+ }
+ /*
+ * path is a data source URI that can appear as a file name, a directory name or a wildcard
+ * returns the data fragments - identifiers of data and a list of available hosts
+ */
+ @Override
+ public List<Fragment> getFragments() throws Exception {
+ String localhostname = java.net.InetAddress.getLocalHost().getHostName();
+ String[] localHosts = new String[]{localhostname, localhostname};
+ fragments.add(new Fragment(inputData.getDataSource() + ".1" /* source name */,
+ localHosts /* available hosts list */,
+ "fragment1".getBytes()));
+ fragments.add(new Fragment(inputData.getDataSource() + ".2" /* source name */,
+ localHosts /* available hosts list */,
+ "fragment2".getBytes()));
+ fragments.add(new Fragment(inputData.getDataSource() + ".3" /* source name */,
+ localHosts /* available hosts list */,
+ "fragment3".getBytes()));
+ return fragments;
+ }
+}
+```
+
+#### Dummy Accessor<a id="dummyaccessor"></a>
+
+``` java
+import org.apache.hawq.pxf.api.WriteAccessor;
+import org.apache.hawq.pxf.api.OneRow;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import org.apache.hawq.pxf.api.utilities.Plugin;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+/*
+ * Internal interface that defines the access to a file on HDFS. All classes
+ * that implement actual access to an HDFS file (sequence file, avro file,...)
+ * must respect this interface
+ * Dummy implementation, for documentation
+ */
+public class DummyAccessor extends Plugin implements ReadAccessor, WriteAccessor {
+ private static final Log LOG = LogFactory.getLog(DummyAccessor.class);
+ private int rowNumber;
+ private int fragmentNumber;
+ public DummyAccessor(InputData metaData) {
+ super(metaData);
+ }
+ @Override
+ public boolean openForRead() throws Exception {
+ /* fopen or similar */
+ return true;
+ }
+ @Override
+ public OneRow readNextObject() throws Exception {
+ /* return next row , <key=fragmentNo.rowNo, val=rowNo,text,fragmentNo>*/
+ /* check for EOF */
+ if (fragmentNumber > 0)
+ return null; /* signal EOF, close will be called */
+ int fragment = inputData.getDataFragment();
+ String fragmentMetadata = new String(inputData.getFragmentMetadata());
+ /* generate row */
+ OneRow row = new OneRow(fragment + "." + rowNumber, /* key */
+ rowNumber + "," + fragmentMetadata + "," + fragment /* value */);
+ /* advance */
+ rowNumber += 1;
+ rowNumber = 0;
+ fragmentNumber += 1;
+ }
+ /* return data */
+ return row;
+ }
+ @Override
+ public void closeForRead() throws Exception {
+ /* fclose or similar */
+ }
+ @Override
+ public boolean openForWrite() throws Exception {
+ /* fopen or similar */
+ return true;
+ }
+ @Override
+ public boolean writeNextObject(OneRow onerow) throws Exception {
+ LOG.info(onerow.getData());
+ return true;
+ }
+ @Override
+ public void closeForWrite() throws Exception {
+ /* fclose or similar */
+ }
+}
+```
+
+#### Dummy Resolver<a id="dummyresolver"></a>
+
+``` java
+import org.apache.hawq.pxf.api.OneField;
+import org.apache.hawq.pxf.api.OneRow;
+import org.apache.hawq.pxf.api.ReadResolver;
+import org.apache.hawq.pxf.api.WriteResolver;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import org.apache.hawq.pxf.api.utilities.Plugin;
+import java.util.LinkedList;
+import java.util.List;
+import static org.apache.hawq.pxf.api.io.DataType.INTEGER;
+import static org.apache.hawq.pxf.api.io.DataType.VARCHAR;
+
+/*
+ * Class that defines the deserializtion of one record brought from the external input data.
+ * Every implementation of a deserialization method (Writable, Avro, BP, Thrift, ...)
+ * must inherit this abstract class
+ * Dummy implementation, for documentation
+ */
+public class DummyResolver extends Plugin implements ReadResolver, WriteResolver {
+ private int rowNumber;
+ public DummyResolver(InputData metaData) {
+ super(metaData);
+ rowNumber = 0;
+ }
+ @Override
+ public List<OneField> getFields(OneRow row) throws Exception {
+ /* break up the row into fields */
+ List<OneField> output = new LinkedList<OneField>();
+ String[] fields = ((String) row.getData()).split(",");
+ output.add(new OneField(INTEGER.getOID() /* type */, Integer.parseInt(fields[0]) /* value */));
+ output.add(new OneField(VARCHAR.getOID(), fields[1]));
+ output.add(new OneField(INTEGER.getOID(), Integer.parseInt(fields[2])));
+ return output;
+ }
+ @Override
+ public OneRow setFields(List<OneField> record) throws Exception {
+ /* should read inputStream row by row */
+ return rowNumber > 5
+ ? null
+ : new OneRow(null, "row number " + rowNumber++);
+ }
+}
+```
+
+#### Usage Example<a id="usageexample"></a>
+
+``` sql
+psql=# CREATE EXTERNAL TABLE dummy_tbl
+ (int1 integer, word text, int2 integer)
+LOCATION ('pxf://localhost:51200/dummy_location'
+ '?FRAGMENTER=DummyFragmenter'
+ '&ACCESSOR=DummyAccessor'
+ '&RESOLVER=DummyResolver')
+FORMAT 'custom' (formatter = 'pxfwritable_import');
+
+CREATE EXTERNAL TABLE
+psql=# SELECT * FROM dummy_tbl;
+int1 | word | int2
+------+------+------
+0 | fragment1 | 0
+1 | fragment1 | 0
+0 | fragment2 | 0
+1 | fragment2 | 0
+0 | fragment3 | 0
+1 | fragment3 | 0
+(6 rows)
+
+psql=# CREATE WRITABLE EXTERNAL TABLE dummy_tbl_write
+ (int1 integer, word text, int2 integer)
+LOCATION ('pxf://localhost:51200/dummy_location'
+ '?ACCESSOR=DummyAccessor'
+ '&RESOLVER=DummyResolver')
+FORMAT 'custom' (formatter = 'pxfwritable_export');
+
+CREATE EXTERNAL TABLE
+psql=# INSERT INTO dummy_tbl_write VALUES (1, 'a', 11), (2, 'b', 22);
+INSERT 0 2
+```
+
+## Renamed Package Reference<a id="topic_b44_yw4_c5"></a>
+
+Package names have been renamed to reflect the move of PXF to Apache HAWQ.
+
+If you have any customized PXF plug-ins, you must recompile your plug-in to use the new package names for PXF APIs. If you have defined any PXF tables with built-in plug-ins, you must recreate the tables with the new package names. Using the PXF `PROFILE` option eases this task.
+
+| Old Name | New Name |
+|--------------------------------------------------------|------------------------------------------------------------|
+| com.pivotal.pxf.plugins.hdfs.LineBreakAccessor | org.apache.hawq.pxf.plugins.hdfs.LineBreakAccessor |
+| com.pivotal.pxf.plugins.hdfs.QuotedLineBreakAccessor | org.apache.hawq.pxf.plugins.hdfs.QuotedLineBreakAccessor |
+| com.pivotal.pxf.plugins.hdfs.AvroFileAccessor | org.apache.hawq.pxf.plugins.hdfs.AvroFileAccessor |
+| com.pivotal.pxf.plugins.hdfs.SequenceFileAccessor | org.apache.hawq.pxf.plugins.hdfs.SequenceFileAccessor |
+| com.pivotal.pxf.plugins.hdfs.StringPassResolver | org.apache.hawq.pxf.plugins.hdfs.StringPassResolver |
+| com.pivotal.pxf.plugins.hdfs.AvroResolver | org.apache.hawq.pxf.plugins.hdfs.AvroResolver |
+| com.pivotal.pxf.plugins.hdfs.WritableResolver | org.apache.hawq.pxf.plugins.hdfs.WritableResolver |
+| com.pivotal.pxf.plugins.hdfs.HdfsDataFragmenter | org.apache.hawq.pxf.plugins.hdfs.HdfsDataFragmenter |
+| com.pivotal.pxf.plugins.hbase.HBaseAccessor | org.apache.hawq.pxf.plugins.hbase.HBaseAccessor |
+| com.pivotal.pxf.plugins.hbase.HBaseDataFragmenter | org.apache.hawq.pxf.plugins.hbase.HBaseDataFragmenter |
+| com.pivotal.pxf.plugins.hbase.HBaseResolver | org.apache.hawq.pxf.plugins.hbase.HBaseResolver |
+| com.pivotal.pxf.plugins.hive.HiveAccessor | org.apache.hawq.pxf.plugins.hive.HiveAccessor |
+| com.pivotal.pxf.plugins.hive.HiveColumnarSerdeResolver | org.apache.hawq.pxf.plugins.hive.HiveColumnarSerdeResolver |
+| com.pivotal.pxf.plugins.hive.HiveDataFragmenter | org.apache.hawq.pxf.plugins.hive.HiveDataFragmenter |
+| com.pivotal.pxf.plugins.hive.HiveInputFormatFragmenter | org.apache.hawq.pxf.plugins.hive.HiveInputFormatFragmenter |
+| com.pivotal.pxf.plugins.hive.HiveLineBreakAccessor | org.apache.hawq.pxf.plugins.hive.HiveLineBreakAccessor |
+| com.pivotal.pxf.plugins.hive.HiveRCFileAccessor | org.apache.hawq.pxf.plugins.hive.HiveRCFileAccessor |
+| com.pivotal.pxf.plugins.hive.HiveResolver | org.apache.hawq.pxf.plugins.hive.HiveResolver |
+| com.pivotal.pxf.plugins.hive.HiveStringPassResolver | org.apache.hawq.pxf.plugins.hive.HiveStringPassResolver |
+
+
diff --git a/pxf/PivotalExtensionFrameworkPXF.html.md.erb b/pxf/PivotalExtensionFrameworkPXF.html.md.erb
new file mode 100644
index 0000000..b705984
--- /dev/null
+++ b/pxf/PivotalExtensionFrameworkPXF.html.md.erb
@@ -0,0 +1,41 @@
+---
+title: Working with PXF and External Data
+---
+
+HAWQ Extension Framework (PXF) is an extensible framework that allows HAWQ to query external system data.
+
+PXF includes built-in connectors for accessing data inside HDFS files, Hive tables, and HBase tables. PXF also integrates with HCatalog to query Hive tables directly.
+
+PXF allows users to create custom connectors to access other parallel data stores or processing engines. To create these connectors using Java plug-ins, see the [PXF External Tables and API](PXFExternalTableandAPIReference.html).
+
+- **[Installing PXF Plug-ins](../pxf/InstallPXFPlugins.html)**
+
+ This topic describes how to install the built-in PXF service plug-ins that are required to connect PXF to HDFS, Hive, and HBase. You should install the appropriate RPMs on each node in your cluster.
+
+- **[Configuring PXF](../pxf/ConfigurePXF.html)**
+
+ This topic describes how to configure the PXF service.
+
+- **[Accessing HDFS File Data](../pxf/HDFSFileDataPXF.html)**
+
+ This topic describes how to access HDFS file data using PXF.
+
+- **[Accessing Hive Data](../pxf/HivePXF.html)**
+
+ This topic describes how to access Hive data using PXF. You have several options for querying data stored in Hive. You can create external tables in PXF and then query those tables, or you can easily query Hive tables by using HAWQ and PXF's integration with HCatalog. HAWQ accesses Hive table metadata stored in HCatalog.
+
+- **[Accessing HBase Data](../pxf/HBasePXF.html)**
+
+ This topic describes how to access HBase data using PXF.
+
+- **[Using Profiles to Read and Write Data](../pxf/ReadWritePXF.html)**
+
+ PXF profiles are collections of common metadata attributes that can be used to simplify the reading and writing of data. You can use any of the built-in profiles that come with PXF or you can create your own.
+
+- **[PXF External Tables and API](../pxf/PXFExternalTableandAPIReference.html)**
+
+ You can use the PXF API to create your own connectors to access any other type of parallel data store or processing engine.
+
+- **[Troubleshooting PXF](../pxf/TroubleshootingPXF.html)**
+
+
diff --git a/pxf/ReadWritePXF.html.md.erb b/pxf/ReadWritePXF.html.md.erb
new file mode 100644
index 0000000..f206cf5
--- /dev/null
+++ b/pxf/ReadWritePXF.html.md.erb
@@ -0,0 +1,132 @@
+---
+title: Using Profiles to Read and Write Data
+---
+
+PXF profiles are collections of common metadata attributes that can be used to simplify the reading and writing of data. You can use any of the built-in profiles that come with PXF or you can create your own.
+
+For example, if you are writing single line records to text files on HDFS, you could use the built-in HdfsTextSimple profile. You specify this profile when you create the PXF external table used to write the data to HDFS.
+
+## Built-In Profiles<a id="built-inprofiles"></a>
+
+PXF comes with a number of built-in profiles that group together a collection of metadata attributes. PXF built-in profiles simplify access to the following types of data storage systems:
+
+- HDFS File Data (Read + Write)
+- Hive (Read only)
+- HBase (Read only)
+
+You can specify a built-in profile when you want to read data that exists inside HDFS files, Hive tables, HBase tables, and for writing data into HDFS files.
+
+<table>
+<colgroup>
+<col width="33%" />
+<col width="33%" />
+<col width="33%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Profile</th>
+<th>Description</th>
+<th>Fragmenter/Accessor/Resolver</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>HdfsTextSimple</td>
+<td>Read or write delimited single line records from or to plain text files on HDFS.</td>
+<td><ul>
+<li>org.apache.hawq.pxf.plugins.hdfs.HdfsDataFragmenter</li>
+<li>org.apache.hawq.pxf.plugins.hdfs.LineBreakAccessor</li>
+<li>org.apache.hawq.pxf.plugins.hdfs.StringPassResolver</li>
+</ul></td>
+</tr>
+<tr class="even">
+<td>HdfsTextMulti</td>
+<td>Read delimited single or multi-line records (with quoted linefeeds) from plain text files on HDFS. This profile is not splittable (non parallel); therefore reading is slower than reading with HdfsTextSimple.</td>
+<td><ul>
+<li>org.apache.hawq.pxf.plugins.hdfs.HdfsDataFragmenter</li>
+<li>org.apache.hawq.pxf.plugins.hdfs.QuotedLineBreakAccessor</li>
+<li>org.apache.hawq.pxf.plugins.hdfs.StringPassResolver</li>
+</ul></td>
+</tr>
+<tr class="odd">
+<td>Hive</td>
+<td>Use this when connecting to Hive. The Hive table can use any of the available storage formats: text, RC, ORC, Sequence, or Parquet.</td>
+<td><ul>
+<li>org.apache.hawq.pxf.plugins.hive.HiveDataFragmenter</li>
+<li>org.apache.hawq.pxf.plugins.hive.HiveAccessor</li>
+<li>org.apache.hawq.pxf.plugins.hive.HiveResolver</li>
+</ul></td>
+</tr>
+<tr class="even">
+<td>HiveRC</td>
+<td>Use this when connecting to a Hive table where each partition is stored as an RCFile. This profile is optimized for it.
+<div class="note note">
+Note: The <code class="ph codeph">DELIMITER</code> parameter is mandatory.
+</div></td>
+<td><ul>
+<li>org.apache.hawq.pxf.plugins.hive.HiveInputFormatFragmenter</li>
+<li>org.apache.hawq.pxf.plugins.hive.HiveRCFileAccessor</li>
+<li>org.apache.hawq.pxf.plugins.hive.HiveColumnarSerdeResolver</li>
+</ul></td>
+</tr>
+<tr class="odd">
+<td>HiveText</td>
+<td>Use this profile when connecting to a Hive table where each partition is stored as a text file. This profile is optimized for it.
+<div class="note note">
+Note: The <code class="ph codeph">DELIMITER</code> parameter is mandatory.
+</div></td>
+<td><ul>
+<li>org.apache.hawq.pxf.plugins.hive.HiveInputFormatFragmenter</li>
+<li>org.apache.hawq.pxf.plugins.hive.HiveLineBreakAccessor</li>
+<li>org.apache.hawq.pxf.plugins.hive.HiveStringPassResolver</li>
+</ul></td>
+</tr>
+<tr class="even">
+<td>HBase</td>
+<td>Use this profile when connected to an HBase data store engine.</td>
+<td><ul>
+<li>org.apache.hawq.pxf.plugins.hbase.HBaseDataFragmenter</li>
+<li>org.apache.hawq.pxf.plugins.hbase.HBaseAccessor</li>
+<li>org.apache.hawq.pxf.plugins.hbase.HBaseResolver</li>
+</ul></td>
+</tr>
+<tr class="odd">
+<td>Avro</td>
+<td>Use this profile for reading Avro files (fileName.avro).</td>
+<td><ul>
+<li>org.apache.hawq.pxf.plugins.hdfs.HdfsDataFragmenter</li>
+<li>org.apache.hawq.pxf.plugins.hdfs.AvroFileAccessor</li>
+<li>org.apache.hawq.pxf.plugins.hdfs.AvroResolver</li>
+</ul></td>
+</tr>
+</tbody>
+</table>
+
+## Adding and Updating Profiles<a id="addingandupdatingprofiles"></a>
+
+Administrators can add new profiles or edit the built-in profiles inside `/etc/conf/pxf-profiles.xml`. You can use all the profiles in `/etc/conf/pxf-profiles.xml`.
+
+**Note:** Add any JAR files that contain custom profile plug-ins to the `/etc/pxf/conf/pxf-public.classpath` configuration file.
+
+Each profile has a mandatory unique name and an optional description.
+
+In addition, each profile contains a set of plug-ins that are an extensible set of metadata attributes.
+
+After you make changes in `pxf-profiles.xml` (or any other PXF configuration file), propagate the changes to all nodes with PXF installed, and then restart the PXF service on all nodes.
+
+## Custom Profile Example<a id="customprofileexample"></a>
+
+``` xml
+<profile>
+ <name>MyCustomProfile</name>
+ <description>A Custom Profile Example</description>
+ <plugins>
+ <fragmenter>package.name.CustomProfileFragmenter</fragmenter>
+ <accessor>package.name.CustomProfileAccessor</accessor>
+ <customPlugin1>package.name.MyCustomPluginValue1</customPlugin1>
+ <customPlugin2>package.name.MyCustomPluginValue2</customPlugin2>
+ </plugins>
+</profile>
+```
+
+
diff --git a/pxf/TroubleshootingPXF.html.md.erb b/pxf/TroubleshootingPXF.html.md.erb
new file mode 100644
index 0000000..a114ef0
--- /dev/null
+++ b/pxf/TroubleshootingPXF.html.md.erb
@@ -0,0 +1,118 @@
+---
+title: Troubleshooting PXF
+---
+
+The following table describes some common errors while using PXF:
+
+<table>
+<caption><span class="tablecap">Table 1. PXF Errors and Explanation</span></caption>
+<colgroup>
+<col width="50%" />
+<col width="50%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Error</th>
+<th>Common Explanation</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>ERROR: invalid URI pxf://localhost:51200/demo/file1: missing options section</td>
+<td><code class="ph codeph">LOCATION</code> does not include options after the file name: <code class="ph codeph"><path>?<key>=<value>&<key>=<value>...</code></td>
+</tr>
+<tr class="even">
+<td>ERROR: protocol "pxf" does not exist</td>
+<td>HAWQ is not compiled with PXF protocol. It requires the GPSQL version of HAWQ</td>
+</tr>
+<tr class="odd">
+<td>ERROR: remote component error (0) from '<x>': There is no pxf servlet listening on the host and port specified in the external table url.</td>
+<td>Wrong server or port, or the service is not started</td>
+</tr>
+<tr class="even">
+<td>ERROR: Missing FRAGMENTER option in the pxf uri: pxf://localhost:51200/demo/file1?a=a</td>
+<td>No <code class="ph codeph">FRAGMENTER</code> option was specified in <code class="ph codeph">LOCATION</code>.</td>
+</tr>
+<tr class="odd">
+<td>ERROR: remote component error (500) from '<x>': type Exception report message org.apache.hadoop.mapred.InvalidInputException:
+<p>Input path does not exist: hdfs://0.0.0.0:8020/demo/file1 </p></td>
+<td>File or pattern given in <code class="ph codeph">LOCATION</code> doesn't exist on specified path.</td>
+</tr>
+<tr class="even">
+<td>ERROR: remote component error (500) from '<x>': type Exception report message org.apache.hadoop.mapred.InvalidInputException : Input Pattern hdfs://0.0.0.0:8020/demo/file* matches 0 files </td>
+<td>File or pattern given in <code class="ph codeph">LOCATION</code> doesn't exist on specified path.</td>
+</tr>
+<tr class="odd">
+<td>ERROR: remote component error (500) from '<x>': PXF not correctly installed in CLASSPATH</td>
+<td>Cannot find PXF Jar</td>
+</tr>
+<tr class="even">
+<td>ERROR: PXF API encountered a HTTP 404 error. Either the PXF service (tomcat) on data node was not started or PXF webapp was not started.</td>
+<td>Either the required data node does not exist or PXF service (tcServer) on data node is not started or PXF webapp was not started</td>
+</tr>
+<tr class="odd">
+<td>ERROR: remote component error (500) from '<x>': type Exception report message java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/client/HTableInterface</td>
+<td>One of the classes required for running PXF or one of its plug-ins is missing. Check that all resources in the PXF classpath files exist on the cluster nodes</td>
+</tr>
+<tr class="even">
+<td>ERROR: remote component error (500) from '<x>': type Exception report message java.io.IOException: Can't get Master Kerberos principal for use as renewer</td>
+<td>Secure PXF: YARN isn't properly configured for secure (Kerberized) HDFS installs</td>
+</tr>
+<tr class="odd">
+<td>ERROR: fail to get filesystem credential for uri hdfs://<namenode>:8020/</td>
+<td>Secure PXF: Wrong HDFS host or port is not 8020 (this is a limitation that will be removed in the next release)</td>
+</tr>
+<tr class="even">
+<td>ERROR: remote component error (413) from '<x>': HTTP status code is 413 but HTTP response string is empty</td>
+<td>The PXF table number of attributes and their name sizes are too large for tcServer to accommodate in its request buffer. The solution is to increase the value of the maxHeaderCount and maxHttpHeaderSize parameters on server.xml on tcServer instance on all nodes and then restart PXF:
+<p><Connector acceptCount="100" connectionTimeout="20000" executor="tomcatThreadPool" maxKeepAliveRequests="15"maxHeaderCount="<some larger value>"maxHttpHeaderSize="<some larger value in bytes>" port="${bio.http.port}" protocol="org.apache.coyote.http11.Http11Protocol" redirectPort="${bio.https.port}"/></p></td>
+</tr>
+<tr class="odd">
+<td>ERROR: remote component error (500) from '<x>': type Exception report message java.lang.Exception: Class com.pivotal.pxf.<plugin name> does not appear in classpath. Plugins provided by PXF must start with "org.apache.hawq.pxf"</td>
+<td>Querying a PXF table that still uses the old package name ("com.pivotal.pxf.*") results in an error message that recommends moving to the new package name ("org.apache.hawq.pxf"). See [Renamed Package Reference](PXFExternalTableandAPIReference.html#topic_b44_yw4_c5).</td>
+</tr>
+<tr class="even">
+<td><strong>HBase Specific Errors</strong></td>
+<td> </td>
+</tr>
+<tr class="odd">
+<td>ERROR: remote component error (500) from '<x>': type Exception report message org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find region for t1,,99999999999999 after 10 tries.</td>
+<td>HBase service is down, probably HRegionServer</td>
+</tr>
+<tr class="even">
+<td>ERROR: remote component error (500) from '<x>': type Exception report message org.apache.hadoop.hbase.TableNotFoundException: nosuch</td>
+<td>HBase cannot find the requested table</td>
+</tr>
+<tr class="odd">
+<td>ERROR: remote component error (500) from '<x>': type Exception report message java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/client/HTableInterface</td>
+<td>PXF cannot find a required JAR file, probably HBase's</td>
+</tr>
+<tr class="even">
+<td>ERROR: remote component error (500) from '<x>': type Exception report message java.lang.NoClassDefFoundError: org/apache/zookeeper/KeeperException</td>
+<td>PXF cannot find ZooKeeper's JAR</td>
+</tr>
+<tr class="odd">
+<td>ERROR: remote component error (500) from '<x>': type Exception report message java.lang.Exception: java.lang.IllegalArgumentException: Illegal HBase column name a, missing :</td>
+<td>PXF table has an illegal field name. Each field name must correspond to an HBase column in the syntax <column family>:<field name></td>
+</tr>
+<tr class="even">
+<td>ERROR: remote component error (500) from '<x>': type Exception report message org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column family a does not exist in region t1,,1405517248353.85f4977bfa88f4d54211cb8ac0f4e644. in table 't1', {NAME =&gt; 'cf', DATA_BLOCK_ENCODING =&gt; 'NONE', BLOOMFILTER =&gt; 'ROW', REPLICATION_SCOPE =&gt; '0', COMPRESSION =&gt; 'NONE', VERSIONS =&gt; '1', TTL =&gt; '2147483647', MIN_VERSIONS =&gt; '0', KEEP_DELETED_CELLS =&gt; 'false', BLOCKSIZE =&gt; '65536', ENCODE_ON_DISK =&gt; 'true', IN_MEMORY =&gt; 'false', BLOCKCACHE =&gt; 'true'}</td>
+<td>Required HBase table does not contain the requested column</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">Hive Specific Errors</code></td>
+<td> </td>
+</tr>
+<tr class="even">
+<td>ERROR: remote component error (500) from '<x>': type Exception report message java.lang.RuntimeException: Failed to connect to Hive metastore: java.net.ConnectException: Connection refused</td>
+<td>Hive Metastore service is down</td>
+</tr>
+<tr class="odd">
+<td>ERROR: remote component error (500) from '<x>': type Exception report message
+<p>NoSuchObjectException(message:default.players table not found)</p></td>
+<td>Table doesn't exist in Hive</td>
+</tr>
+</tbody>
+</table>
+
+
diff --git a/query/HAWQQueryProcessing.html.md.erb b/query/HAWQQueryProcessing.html.md.erb
new file mode 100644
index 0000000..ace1df4
--- /dev/null
+++ b/query/HAWQQueryProcessing.html.md.erb
@@ -0,0 +1,60 @@
+---
+title: About HAWQ Query Processing
+---
+
+This topic provides an overview of how HAWQ processes queries. Understanding this process can be useful when writing and tuning queries.
+
+Users issue queries to HAWQ as they would to any database management system. They connect to the database instance on the HAWQ master host using a client application such as `psql` and submit SQL statements.
+
+## Understanding Query Planning and Dispatch<a id="topic2"></a>
+
+After a query is accepted on master, the master parses and analyzes the query. After completing its analysis, the master generates a query tree and provides the query tree to the query optimizer.
+
+The query optimizer generates a query plan. Given the cost information of the query plan, resources are requested from the HAWQ resource manager. After the resources are obtained, the dispatcher starts virtual segments and dispatches the query plan to virtual segments for execution.
+
+This diagram depicts basic query flow in HAWQ.
+
+<img src="../images/basic_query_flow.png" id="topic2__image_ezs_wbh_sv" class="image" width="672" />
+
+## Understanding HAWQ Query Plans<a id="topic3"></a>
+
+A query plan is the set of operations HAWQ will perform to produce the answer to a query. Each *node* or step in the plan represents a database operation such as a table scan, join, aggregation, or sort. Plans are read and executed from bottom to top.
+
+In addition to common database operations such as tables scans, joins, and so on, HAWQ has an additional operation type called *motion*. A motion operation involves moving tuples between the segments during query processing. Note that not every query requires a motion. For example, a targeted query plan does not require data to move across the interconnect.
+
+To achieve maximum parallelism during query execution, HAWQ divides the work of the query plan into *slices*. A slice is a portion of the plan that segments can work on independently. A query plan is sliced wherever a *motion* operation occurs in the plan, with one slice on each side of the motion.
+
+For example, consider the following simple query involving a join between two tables:
+
+``` sql
+SELECT customer, amount
+FROM sales JOIN customer USING (cust_id)
+WHERE dateCol = '04-30-2008';
+```
+
+[Query Slice Plan](#topic3__iy140224) shows the query plan. Each segment receives a copy of the query plan and works on it in parallel.
+
+The query plan for this example has a *redistribute motion* that moves tuples between the segments to complete the join. The redistribute motion is necessary because the customer table is distributed across the segments by `cust_id`, but the sales table is distributed across the segments by `sale_id`. To perform the join, the `sales` tuples must be redistributed by `cust_id`. The plan is sliced on either side of the redistribute motion, creating *slice 1* and *slice 2*.
+
+This query plan has another type of motion operation called a *gather motion*. A gather motion is when the segments send results back up to the master for presentation to the client. Because a query plan is always sliced wherever a motion occurs, this plan also has an implicit slice at the very top of the plan (*slice 3*). Not all query plans involve a gather motion. For example, a `CREATE TABLE x AS SELECT...` statement would not have a gather motion because tuples are sent to the newly created table, not to the master.
+
+<a id="topic3__iy140224"></a>
+<span class="figtitleprefix">Figure: </span>Query Slice Plan
+
+<img src="../images/slice_plan.jpg" class="image" width="462" height="382" />
+
+## Understanding Parallel Query Execution<a id="topic4"></a>
+
+HAWQ creates a number of database processes to handle the work of a query. On the master, the query worker process is called the *query dispatcher* (QD). The QD is responsible for creating and dispatching the query plan. It also accumulates and presents the final results. On virtual segments, a query worker process is called a *query executor* (QE). A QE is responsible for completing its portion of work and communicating its intermediate results to the other worker processes.
+
+There is at least one worker process assigned to each *slice* of the query plan. A worker process works on its assigned portion of the query plan independently. During query execution, each virtual segment will have a number of processes working on the query in parallel.
+
+Related processes that are working on the same slice of the query plan but on different virtual segments are called *gangs*. As a portion of work is completed, tuples flow up the query plan from one gang of processes to the next. This inter-process communication between virtual segments is referred to as the *interconnect* component of HAWQ.
+
+[Query Worker Processes](#topic4__iy141495) shows the query worker processes on the master and two virtual segment instances for the query plan illustrated in [Query Slice Plan](#topic3__iy140224).
+
+<a id="topic4__iy141495"></a>
+<span class="figtitleprefix">Figure: </span>Query Worker Processes
+
+<img src="../images/gangs.jpg" class="image" width="318" height="288" />
+
diff --git a/query/defining-queries.html.md.erb b/query/defining-queries.html.md.erb
new file mode 100644
index 0000000..804171d
--- /dev/null
+++ b/query/defining-queries.html.md.erb
@@ -0,0 +1,528 @@
+---
+title: Defining Queries
+---
+
+HAWQ is based on the PostgreSQL implementation of the SQL standard. SQL commands are typically entered using the standard PostgreSQL interactive terminal `psql`, but other programs that have similar functionality can be used as well.
+
+
+## SQL Lexicon<a id="topic3"></a>
+
+SQL is a standard language for accessing databases. The language consists of elements that enable data storage, retrieval, analysis, viewing, and so on. You use SQL commands to construct queries and commands that the HAWQ engine understands.
+
+SQL queries consist of a sequence of commands. Commands consist of a sequence of valid tokens in correct syntax order, terminated by a semicolon (`;`).
+
+H uses PostgreSQL's structure and syntax, with some exceptions. For more information about SQL rules and concepts in PostgreSQL, see "SQL Syntax" in the PostgreSQL documentation.
+
+## SQL Value Expressions<a id="topic4"></a>
+
+SQL value expressions consist of one or more values, symbols, operators, SQL functions, and data. The expressions compare data or perform calculations and return a value as the result. Calculations include logical, arithmetic, and set operations.
+
+The following are value expressions:
+
+- Aggregate expressions
+- Array constructors
+- Column references
+- Constant or literal values
+- Correlated subqueries
+- Field selection expressions
+- Function calls
+- New column values in an `INSERT`
+- Operator invocation column references
+- Positional parameter references, in the body of a function definition or prepared statement
+- Row constructors
+- Scalar subqueries
+- Search conditions in a `WHERE` clause
+- Target lists of a `SELECT` command
+- Type casts
+- Value expressions in parentheses, useful to group sub-expressions and override precedence
+- Window expressions
+
+SQL constructs such as functions and operators are expressions but do not follow any general syntax rules. For more information about these constructs, see [Using Functions and Operators](functions-operators.html#topic26).
+
+### Column References<a id="topic5"></a>
+
+A column reference has the form:
+
+```
+correlation.columnname
+```
+
+Here, `correlation` is the name of a table (possibly qualified with a schema name) or an alias for a table defined with a `FROM` clause or one of the keywords `NEW` or `OLD`. `NEW` and `OLD` can appear only in rewrite rules, but you can use other correlation names in any SQL statement. If the column name is unique across all tables in the query, you can omit the "`correlation.`" part of the column reference.
+
+### Positional Parameters<a id="topic6"></a>
+
+Positional parameters are arguments to SQL statements or functions that you reference by their positions in a series of arguments. For example, `$1` refers to the first argument, `$2` to the second argument, and so on. The values of positional parameters are set from arguments external to the SQL statement or supplied when SQL functions are invoked. Some client libraries support specifying data values separately from the SQL command, in which case parameters refer to the out-of-line data values. A parameter reference has the form:
+
+```
+$number
+```
+
+For example:
+
+``` pre
+CREATE FUNCTION dept(text) RETURNS dept
+ AS $$ SELECT * FROM dept WHERE name = $1 $$
+ LANGUAGE SQL;
+```
+
+Here, the `$1` references the value of the first function argument whenever the function is invoked.
+
+### Subscripts<a id="topic7"></a>
+
+If an expression yields a value of an array type, you can extract a specific element of the array value as follows:
+
+``` pre
+expression[subscript]
+```
+
+You can extract multiple adjacent elements, called an array slice, as follows (including the brackets):
+
+``` pre
+expression[lower_subscript:upper_subscript]
+```
+
+Each subscript is an expression and yields an integer value.
+
+Array expressions usually must be in parentheses, but you can omit the parentheses when the expression to be subscripted is a column reference or positional parameter. You can concatenate multiple subscripts when the original array is multidimensional. For example (including the parentheses):
+
+``` pre
+mytable.arraycolumn[4]
+```
+
+``` pre
+mytable.two_d_column[17][34]
+```
+
+``` pre
+$1[10:42]
+```
+
+``` pre
+(arrayfunction(a,b))[42]
+```
+
+### Field Selections<a id="topic8"></a>
+
+If an expression yields a value of a composite type (row type), you can extract a specific field of the row as follows:
+
+```
+expression.fieldname
+```
+
+The row expression usually must be in parentheses, but you can omit these parentheses when the expression to be selected from is a table reference or positional parameter. For example:
+
+``` pre
+mytable.mycolumn
+```
+
+``` pre
+$1.somecolumn
+```
+
+``` pre
+(rowfunction(a,b)).col3
+```
+
+A qualified column reference is a special case of field selection syntax.
+
+### Operator Invocations<a id="topic9"></a>
+
+Operator invocations have the following possible syntaxes:
+
+``` pre
+expression operator expression(binary infix operator)
+```
+
+``` pre
+operator expression(unary prefix operator)
+```
+
+``` pre
+expression operator(unary postfix operator)
+```
+
+Where *operator* is an operator token, one of the key words `AND`, `OR`, or `NOT`, or qualified operator name in the form:
+
+``` pre
+OPERATOR(schema.operatorname)
+```
+
+Available operators and whether they are unary or binary depends on the operators that the system or user defines. For more information about built-in operators, see [Built-in Functions and Operators](functions-operators.html#topic29).
+
+### Function Calls<a id="topic10"></a>
+
+The syntax for a function call is the name of a function (possibly qualified with a schema name), followed by its argument list enclosed in parentheses:
+
+``` pre
+function ([expression [, expression ... ]])
+```
+
+For example, the following function call computes the square root of 2:
+
+``` pre
+sqrt(2)
+```
+
+### Aggregate Expressions<a id="topic11"></a>
+
+An aggregate expression applies an aggregate function across the rows that a query selects. An aggregate function performs a calculation on a set of values and returns a single value, such as the sum or average of the set of values. The syntax of an aggregate expression is one of the following:
+
+- `aggregate_name ([ , ... ] ) [FILTER (WHERE condition)] ` — operates across all input rows for which the expected result value is non-null. `ALL` is the default.
+- `aggregate_name(ALLexpression[ , ... ] ) [FILTER (WHERE condition)]` — operates identically to the first form because `ALL` is the default
+- `aggregate_name(DISTINCT expression[ , ... ] ) [FILTER (WHERE condition)]` — operates across all distinct non-null values of input rows
+- `aggregate_name(*) [FILTER (WHERE condition)]` — operates on all rows with values both null and non-null. Generally, this form is most useful for the `count(*)` aggregate function.
+
+Where *aggregate\_name* is a previously defined aggregate (possibly schema-qualified) and *expression* is any value expression that does not contain an aggregate expression.
+
+For example, `count(*)` yields the total number of input rows, `count(f1)` yields the number of input rows in which `f1` is <span class="ph">non-null, and </span>`count(distinct f1)` yields the number of distinct non-null values of `f1`.
+
+You can specify a condition with the `FILTER` clause to limit the input rows to the aggregate function. For example:
+
+``` sql
+SELECT count(*) FILTER (WHERE gender='F') FROM employee;
+```
+
+The `WHERE condition` of the `FILTER` clause cannot contain a set-returning function, subquery, window function, or outer reference. If you use a user-defined aggregate function, declare the state transition function as `STRICT` (see `CREATE AGGREGATE`).
+
+For predefined aggregate functions, see [Built-in Functions and Operators](functions-operators.html#topic29). You can also add custom aggregate functions.
+
+HAWQ provides the `MEDIAN` aggregate function, which returns the fiftieth percentile of the `PERCENTILE_CONT` result and special aggregate expressions for inverse distribution functions as follows:
+
+``` sql
+PERCENTILE_CONT(_percentage_) WITHIN GROUP (ORDER BY _expression_)
+```
+
+``` sql
+PERCENTILE_DISC(_percentage_) WITHIN GROUP (ORDER BY _expression_)
+```
+
+Currently you can use only these two expressions with the keyword `WITHIN GROUP`.
+
+#### Limitations of Aggregate Expressions<a id="topic12"></a>
+
+The following are current limitations of the aggregate expressions:
+
+- HAWQ does not support the following keywords: ALL, DISTINCT, FILTER and OVER. See [Advanced Aggregate Functions](functions-operators.html#topic31__in2073121) for more details.
+- An aggregate expression can appear only in the result list or HAVING clause of a SELECT command. It is forbidden in other clauses, such as WHERE, because those clauses are logically evaluated before the results of aggregates form. This restriction applies to the query level to which the aggregate belongs.
+- When an aggregate expression appears in a subquery, the aggregate is normally evaluated over the rows of the subquery. If the aggregate's arguments contain only outer-level variables, the aggregate belongs to the nearest such outer level and evaluates over the rows of that query. The aggregate expression as a whole is then an outer reference for the subquery in which it appears, and the aggregate expression acts as a constant over any one evaluation of that subquery. See [Scalar Subqueries](#topic15) and [Built-in functions and operators](functions-operators.html#topic29__in204913).
+- HAWQ does not support DISTINCT with multiple input expressions.
+
+### Window Expressions<a id="topic13"></a>
+
+Window expressions allow application developers to more easily compose complex online analytical processing (OLAP) queries using standard SQL commands. For example, with window expressions, users can calculate moving averages or sums over various intervals, reset aggregations and ranks as selected column values change, and express complex ratios in simple terms.
+
+A window expression represents the application of a *window function* applied to a *window frame*, which is defined in a special `OVER()` clause. A window partition is a set of rows that are grouped together to apply a window function. Unlike aggregate functions, which return a result value for each group of rows, window functions return a result value for every row, but that value is calculated with respect to the rows in a particular window partition. If no partition is specified, the window function is computed over the complete intermediate result set.
+
+The syntax of a window expression is:
+
+``` pre
+window_function ( [expression [, ...]] ) OVER ( window_specification )
+```
+
+Where *`window_function`* is one of the functions listed in [Window functions](functions-operators.html#topic30__in164369), *`expression`* is any value expression that does not contain a window expression, and *`window_specification`* is:
+
+```
+[window_name]
+[PARTITION BY expression [, ...]]
+[[ORDER BY expression [ASC | DESC | USING operator] [, ...]
+ [{RANGE | ROWS}
+ { UNBOUNDED PRECEDING
+ | expression PRECEDING
+ | CURRENT ROW
+ | BETWEEN window_frame_bound AND window_frame_bound }]]
+```
+
+and where `window_frame_bound` can be one of:
+
+```
+ UNBOUNDED PRECEDING
+ expression PRECEDING
+ CURRENT ROW
+ expression FOLLOWING
+ UNBOUNDED FOLLOWING
+```
+
+A window expression can appear only in the select list of a `SELECT` command. For example:
+
+``` sql
+SELECT count(*) OVER(PARTITION BY customer_id), * FROM sales;
+```
+
+The `OVER` clause differentiates window functions from other aggregate or reporting functions. The `OVER` clause defines the *`window_specification`* to which the window function is applied. A window specification has the following characteristics:
+
+- The `PARTITION BY` clause defines the window partitions to which the window function is applied. If omitted, the entire result set is treated as one partition.
+- The `ORDER BY` clause defines the expression(s) for sorting rows within a window partition. The `ORDER BY` clause of a window specification is separate and distinct from the `ORDER BY` clause of a regular query expression. The `ORDER BY` clause is required for the window functions that calculate rankings, as it identifies the measure(s) for the ranking values. For OLAP aggregations, the `ORDER BY` clause is required to use window frames (the `ROWS` | `RANGE` clause).
+
+**Note:** Columns of data types without a coherent ordering, such as `time`, are not good candidates for use in the `ORDER BY` clause of a window specification. `Time`, with or without a specified time zone, lacks a coherent ordering because addition and subtraction do not have the expected effects. For example, the following is not generally true: `x::time < x::time + '2 hour'::interval`
+
+- The `ROWS/RANGE` clause defines a window frame for aggregate (non-ranking) window functions. A window frame defines a set of rows within a window partition. When a window frame is defined, the window function computes on the contents of this moving frame rather than the fixed contents of the entire window partition. Window frames are row-based (`ROWS`) or value-based (`RANGE`).
+
+### Type Casts<a id="topic14"></a>
+
+A type cast specifies a conversion from one data type to another. HAWQ accepts two equivalent syntaxes for type casts:
+
+``` sql
+CAST ( expression AS type )
+expression::type
+```
+
+The `CAST` syntax conforms to SQL; the syntax with `::` is historical PostgreSQL usage.
+
+A cast applied to a value expression of a known type is a run-time type conversion. The cast succeeds only if a suitable type conversion function is defined. This differs from the use of casts with constants. A cast applied to a string literal represents the initial assignment of a type to a literal constant value, so it succeeds for any type if the contents of the string literal are acceptable input syntax for the data type.
+
+You can usually omit an explicit type cast if there is no ambiguity about the type a value expression must produce; for example, when it is assigned to a table column, the system automatically applies a type cast. The system applies automatic casting only to casts marked "OK to apply implicitly" in system catalogs. Other casts must be invoked with explicit casting syntax to prevent unexpected conversions from being applied without the user's knowledge.
+
+### Scalar Subqueries<a id="topic15"></a>
+
+A scalar subquery is a `SELECT` query in parentheses that returns exactly one row with one column. Do not use a `SELECT` query that returns multiple rows or columns as a scalar subquery. The query runs and uses the returned value in the surrounding value expression. A correlated scalar subquery contains references to the outer query block.
+
+### Correlated Subqueries<a id="topic16"></a>
+
+A correlated subquery (CSQ) is a `SELECT` query with a `WHERE` clause or target list that contains references to the parent outer clause. CSQs efficiently express results in terms of results of another query. HAWQ supports correlated subqueries that provide compatibility with many existing applications. A CSQ is a scalar or table subquery, depending on whether it returns one or multiple rows. HAWQ does not support correlated subqueries with skip-level correlations.
+
+### Correlated Subquery Examples<a id="topic17"></a>
+
+#### Example 1 – Scalar correlated subquery<a id="topic18"></a>
+
+``` sql
+SELECT * FROM t1 WHERE t1.x
+> (SELECT MAX(t2.x) FROM t2 WHERE t2.y = t1.y);
+```
+
+#### Example 2 – Correlated EXISTS subquery<a id="topic19"></a>
+
+``` sql
+SELECT * FROM t1 WHERE
+EXISTS (SELECT 1 FROM t2 WHERE t2.x = t1.x);
+```
+
+HAWQ uses one of the following methods to run CSQs:
+
+- Unnest the CSQ into join operations – This method is most efficient, and it is how HAWQ runs most CSQs, including queries from the TPC-H benchmark.
+- Run the CSQ on every row of the outer query – This method is relatively inefficient, and it is how HAWQ runs queries that contain CSQs in the `SELECT` list or are connected by `OR` conditions.
+
+The following examples illustrate how to rewrite some of these types of queries to improve performance.
+
+#### Example 3 - CSQ in the Select List<a id="topic20"></a>
+
+*Original Query*
+
+``` sql
+SELECT T1.a,
+(SELECT COUNT(DISTINCT T2.z) FROM t2 WHERE t1.x = t2.y) dt2
+FROM t1;
+```
+
+Rewrite this query to perform an inner join with `t1` first and then perform a left join with `t1` again. The rewrite applies for only an equijoin in the correlated condition.
+
+*Rewritten Query*
+
+``` sql
+SELECT t1.a, dt2 FROM t1
+LEFT JOIN
+(SELECT t2.y AS csq_y, COUNT(DISTINCT t2.z) AS dt2
+FROM t1, t2 WHERE t1.x = t2.y
+GROUP BY t1.x)
+ON (t1.x = csq_y);
+```
+
+### Example 4 - CSQs connected by OR Clauses<a id="topic21"></a>
+
+*Original Query*
+
+``` sql
+SELECT * FROM t1
+WHERE
+x > (SELECT COUNT(*) FROM t2 WHERE t1.x = t2.x)
+OR x < (SELECT COUNT(*) FROM t3 WHERE t1.y = t3.y)
+```
+
+Rewrite this query to separate it into two parts with a union on the `OR` conditions.
+
+*Rewritten Query*
+
+``` sql
+SELECT * FROM t1
+WHERE x > (SELECT count(*) FROM t2 WHERE t1.x = t2.x)
+UNION
+SELECT * FROM t1
+WHERE x < (SELECT count(*) FROM t3 WHERE t1.y = t3.y)
+```
+
+To view the query plan, use `EXPLAIN SELECT` or `EXPLAIN ANALYZE SELECT`. Subplan nodes in the query plan indicate that the query will run on every row of the outer query, and the query is a candidate for rewriting. For more information about these statements, see [Query Profiling](query-profiling.html#topic39).
+
+### Advanced Table Functions<a id="topic22"></a>
+
+HAWQ supports table functions with `TABLE` value expressions. You can sort input rows for advanced table functions with an `ORDER BY` clause. You can redistribute them with a `SCATTER BY` clause to specify one or more columns or an expression for which rows with the specified characteristics are available to the same process. This usage is similar to using a `DISTRIBUTED BY` clause when creating a table, but the redistribution occurs when the query runs.
+
+**Note:**
+Based on the distribution of data, HAWQ automatically parallelizes table functions with `TABLE` value parameters over the nodes of the cluster.
+
+### Array Constructors<a id="topic23"></a>
+
+An array constructor is an expression that builds an array value from values for its member elements. A simple array constructor consists of the key word `ARRAY`, a left square bracket `[`, one or more expressions separated by commas for the array element values, and a right square bracket `]`. For example,
+
+``` sql
+SELECT ARRAY[1,2,3+4];
+```
+
+```
+ array
+---------
+ {1,2,7}
+```
+
+The array element type is the common type of its member expressions, determined using the same rules as for `UNION` or `CASE` constructs.
+
+You can build multidimensional array values by nesting array constructors. In the inner constructors, you can omit the keyword `ARRAY`. For example, the following two `SELECT` statements produce the same result:
+
+``` sql
+SELECT ARRAY[ARRAY[1,2], ARRAY[3,4]];
+SELECT ARRAY[[1,2],[3,4]];
+```
+
+```
+ array
+---------------
+ {{1,2},{3,4}}
+```
+
+Since multidimensional arrays must be rectangular, inner constructors at the same level must produce sub-arrays of identical dimensions.
+
+Multidimensional array constructor elements are not limited to a sub-`ARRAY` construct; they are anything that produces an array of the proper kind. For example:
+
+``` sql
+CREATE TABLE arr(f1 int[], f2 int[]);
+INSERT INTO arr VALUES (ARRAY[[1,2],[3,4]],
+ARRAY[[5,6],[7,8]]);
+SELECT ARRAY[f1, f2, '{{9,10},{11,12}}'::int[]] FROM arr;
+```
+
+```
+ array
+------------------------------------------------
+ {{{1,2},{3,4}},{{5,6},{7,8}},{{9,10},{11,12}}}
+```
+
+You can construct an array from the results of a subquery. Write the array constructor with the keyword `ARRAY` followed by a subquery in parentheses. For example:
+
+``` sql
+SELECT ARRAY(SELECT oid FROM pg_proc WHERE proname LIKE 'bytea%');
+```
+
+```
+ ?column?
+-----------------------------------------------------------
+ {2011,1954,1948,1952,1951,1244,1950,2005,1949,1953,2006,31}
+```
+
+The subquery must return a single column. The resulting one-dimensional array has an element for each row in the subquery result, with an element type matching that of the subquery's output column. The subscripts of an array value built with `ARRAY` always begin with `1`.
+
+### Row Constructors<a id="topic24"></a>
+
+A row constructor is an expression that builds a row value (also called a composite value) from values for its member fields. For example,
+
+``` sql
+SELECT ROW(1,2.5,'this is a test');
+```
+
+Row constructors have the syntax `rowvalue.*`, which expands to a list of the elements of the row value, as when you use the syntax `.*` at the top level of a `SELECT` list. For example, if table `t` has columns `f1` and `f2`, the following queries are the same:
+
+``` sql
+SELECT ROW(t.*, 42) FROM t;
+SELECT ROW(t.f1, t.f2, 42) FROM t;
+```
+
+By default, the value created by a `ROW` expression has an anonymous record type. If necessary, it can be cast to a named composite type — either the row type of a table, or a composite type created with `CREATE TYPE AS`. To avoid ambiguity, you can explicitly cast the value if necessary. For example:
+
+``` sql
+CREATE TABLE mytable(f1 int, f2 float, f3 text);
+CREATE FUNCTION getf1(mytable) RETURNS int AS 'SELECT $1.f1'
+LANGUAGE SQL;
+```
+
+In the following query, you do not need to cast the value because there is only one `getf1()` function and therefore no ambiguity:
+
+``` sql
+SELECT getf1(ROW(1,2.5,'this is a test'));
+```
+
+```
+ getf1
+-------
+ 1
+```
+
+``` sql
+CREATE TYPE myrowtype AS (f1 int, f2 text, f3 numeric);
+CREATE FUNCTION getf1(myrowtype) RETURNS int AS 'SELECT
+$1.f1' LANGUAGE SQL;
+```
+
+Now we need a cast to indicate which function to call:
+
+``` sql
+SELECT getf1(ROW(1,2.5,'this is a test'));
+```
+```
+ERROR: function getf1(record) is not unique
+```
+
+``` sql
+SELECT getf1(ROW(1,2.5,'this is a test')::mytable);
+```
+
+```
+ getf1
+-------
+ 1
+```
+
+``` sql
+SELECT getf1(CAST(ROW(11,'this is a test',2.5) AS myrowtype));
+```
+
+```
+ getf1
+-------
+ 11
+```
+
+You can use row constructors to build composite values to be stored in a composite-type table column or to be passed to a function that accepts a composite parameter.
+
+### Expression Evaluation Rules<a id="topic25"></a>
+
+The order of evaluation of subexpressions is undefined. The inputs of an operator or function are not necessarily evaluated left-to-right or in any other fixed order.
+
+If you can determine the result of an expression by evaluating only some parts of the expression, then other subexpressions might not be evaluated at all. For example, in the following expression:
+
+``` sql
+SELECT true OR somefunc();
+```
+
+`somefunc()` would probably not be called at all. The same is true in the following expression:
+
+``` sql
+SELECT somefunc() OR true;
+```
+
+This is not the same as the left-to-right evaluation order that Boolean operators enforce in some programming languages.
+
+Do not use functions with side effects as part of complex expressions, especially in `WHERE` and `HAVING` clauses, because those clauses are extensively reprocessed when developing an execution plan. Boolean expressions (`AND`/`OR`/`NOT` combinations) in those clauses can be reorganized in any manner that Boolean algebra laws allow.
+
+Use a `CASE` construct to force evaluation order. The following example is an untrustworthy way to avoid division by zero in a `WHERE` clause:
+
+``` sql
+SELECT ... WHERE x <> 0 AND y/x > 1.5;
+```
+
+The following example shows a trustworthy evaluation order:
+
+``` sql
+SELECT ... WHERE CASE WHEN x <> 0 THEN y/x > 1.5 ELSE false
+END;
+```
+
+This `CASE` construct usage defeats optimization attempts; use it only when necessary.
+
+
diff --git a/query/functions-operators.html.md.erb b/query/functions-operators.html.md.erb
new file mode 100644
index 0000000..ee3a9ad
--- /dev/null
+++ b/query/functions-operators.html.md.erb
@@ -0,0 +1,428 @@
+---
+title: Using Functions and Operators
+---
+
+HAWQ evaluates functions and operators used in SQL expressions.
+
+## Using Functions in HAWQ<a id="topic27"></a>
+
+In HAWQ, functions can only be run on master.
+
+<a id="topic27__in201681"></a>
+
+<span class="tablecap">Table 1. Functions in HAWQ</span>
+
+
+| Function Type | HAWQ Support | Description | Comments |
+|---------------|--------------------|---------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------|
+| IMMUTABLE | Yes | Relies only on information directly in its argument list. Given the same argument values, always returns the same result. | |
+| STABLE | Yes, in most cases | Within a single table scan, returns the same result for same argument values, but results change across SQL statements. | Results depend on database lookups or parameter values. `current_timestamp` family of functions is `STABLE`; values do not change within an execution. |
+| VOLATILE | Restricted | Function values can change within a single table scan. For example: `random()`, `currval()`, `timeofday()`. | Any function with side effects is volatile, even if its result is predictable. For example: `setval()`. |
+
+HAWQ does not support functions that return a table reference (`rangeFuncs`) or functions that use the `refCursor` datatype.
+
+## User-Defined Functions<a id="topic28"></a>
+
+HAWQ supports user-defined functions. See [Extending SQL](http://www.postgresql.org/docs/8.2/static/extend.html) in the PostgreSQL documentation for more information.
+
+In HAWQ, the shared library files for user-created functions must reside in the same library path location on every host in the HAWQ array (masters and segments).
+
+**Important:**
+HAWQ does not support the following:
+
+- Enhanced table functions
+- PL/Java Type Maps
+
+Use the `CREATE FUNCTION` statement to register user-defined functions that are used as described in [Using Functions in HAWQ](#topic27). By default, user-defined functions are declared as `VOLATILE`, so if your user-defined function is `IMMUTABLE` or `STABLE`, you must specify the correct volatility level when you register your function.
+
+### Function Volatility<a id="functionvolatility"></a>
+
+Every function has a **volatility** classification, with the possibilities being VOLATILE, STABLE, or IMMUTABLE. VOLATILE is the default if the [CREATE FUNCTION](../reference/sql/CREATE-FUNCTION.html) command does not specify a category. The volatility category is a promise to the optimizer about the behavior of the function:
+
+- A VOLATILE function can do anything, including modifying the database. It can return different results on successive calls with the same arguments. The optimizer makes no assumptions about the behavior of such functions. A query using a volatile function will re-evaluate the function at every row where its value is needed.
+- A STABLE function cannot modify the database and is guaranteed to return the same results given the same arguments for all rows within a single statement. This category allows the optimizer to optimize multiple calls of the function to a single call.
+- An IMMUTABLE function cannot modify the database and is guaranteed to return the same results given the same arguments forever. This category allows the optimizer to pre-evaluate the function when a query calls it with constant arguments. For example, a query like SELECT ... WHERE x = 2 + 2 can be simplified on sight to SELECT ... WHERE x = 4, because the function underlying the integer addition operator is marked IMMUTABLE.
+
+For best optimization results, you should label your functions with the strictest volatility category that is valid for them.
+
+Any function with side-effects must be labeled VOLATILE, so that calls to it cannot be optimized away. Even a function with no side-effects needs to be labeled VOLATILE if its value can change within a single query; some examples are random(), currval(), timeofday().
+
+Another important example is that the `current_timestamp` family of functions qualify as STABLE, since their values do not change within a transaction.
+
+There is relatively little difference between STABLE and IMMUTABLE categories when considering simple interactive queries that are planned and immediately executed: it doesn't matter a lot whether a function is executed once during planning or once during query execution startup. But there is a big difference if the plan is saved and reused later. Labeling a function IMMUTABLE when it really isn't might allow it to be prematurely folded to a constant during planning, resulting in a stale value being re-used during subsequent uses of the plan. This is a hazard when using prepared statements or when using function languages that cache plans (such as PL/pgSQL).
+
+For functions written in SQL or in any of the standard procedural languages, there is a second important property determined by the volatility category, namely the visibility of any data changes that have been made by the SQL command that is calling the function. A VOLATILE function will see such changes, a STABLE or IMMUTABLE function will not. STABLE and IMMUTABLE functions use a snapshot established as of the start of the calling query, whereas VOLATILE functions obtain a fresh snapshot at the start of each query they execute.
+
+Because of this snapshotting behavior, a function containing only SELECT commands can safely be marked STABLE, even if it selects from tables that might be undergoing modifications by concurrent queries. PostgreSQL will execute all commands of a STABLE function using the snapshot established for the calling query, and so it will see a fixed view of the database throughout that query.
+
+The same snapshotting behavior is used for SELECT commands within IMMUTABLE functions. It is generally unwise to select from database tables within an IMMUTABLE function at all, since the immutability will be broken if the table contents ever change. However, PostgreSQL does not enforce that you do not do that.
+
+A common error is to label a function IMMUTABLE when its results depend on a configuration parameter. For example, a function that manipulates timestamps might well have results that depend on the timezone setting. For safety, such functions should be labeled STABLE instead.
+
+When you create user defined functions, avoid using fatal errors or destructive calls. HAWQ may respond to such errors with a sudden shutdown or restart.
+
+## User Defined Types<a id="userdefinedtypes"></a>
+
+HAWQ can be extended to support new data types. This section describes how to define new base types, which are data types defined below the level of the SQL language. Creating a new base type requires implementing functions to operate on the type in a low-level language, usually C.
+
+A user-defined type must always have input and output functions. These functions determine how the type appears in strings (for input by the user and output to the user) and how the type is organized in memory. The input function takes a null-terminated character string as its argument and returns the internal (in memory) representation of the type. The output function takes the internal representation of the type as argument and returns a null-terminated character string. If we want to do anything more with the type than merely store it, we must provide additional functions to implement whatever operations we'd like to have for the type.
+
+You should be careful to make the input and output functions inverses of each other. If you do not, you will have severe problems when you need to dump your data into a file and then read it back in. This is a particularly common problem when floating-point numbers are involved.
+
+Optionally, a user-defined type can provide binary input and output routines. Binary I/O is normally faster but less portable than textual I/O. As with textual I/O, it is up to you to define exactly what the external binary representation is. Most of the built-in data types try to provide a machine-independent binary representation.
+
+Once we have written the I/O functions and compiled them into a shared library, we can define the complex type in SQL. First we declare it as a shell type:
+
+``` sql
+CREATE TYPE complex;
+```
+
+This serves as a placeholder that allows us to reference the type while defining its I/O functions. Now we can define the I/O functions:
+
+``` sql
+CREATE FUNCTION complex_in(cstring)
+ RETURNS complex
+ AS 'filename'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE FUNCTION complex_out(complex)
+ RETURNS cstring
+ AS 'filename'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE FUNCTION complex_recv(internal)
+ RETURNS complex
+ AS 'filename'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE FUNCTION complex_send(complex)
+ RETURNS bytea
+ AS 'filename'
+ LANGUAGE C IMMUTABLE STRICT;
+```
+
+Finally, we can provide the full definition of the data type:
+
+``` sql
+CREATE TYPE complex (
+ internallength = 16,
+ input = complex_in,
+ output = complex_out,
+ receive = complex_recv,
+ send = complex_send,
+ alignment = double
+);
+```
+
+When you define a new base type, HAWQ automatically provides support for arrays of that type. For historical reasons, the array type has the same name as the base type with the underscore character (\_) prepended.
+
+Once the data type exists, we can declare additional functions to provide useful operations on the data type. Operators can then be defined atop the functions, and if needed, operator classes can be created to support indexing of the data type.
+
+For further details, see the description of the [CREATE TYPE](../reference/sql/CREATE-TYPE.html) command.
+
+## User Defined Operators<a id="userdefinedoperators"></a>
+
+Every operator is "syntactic sugar" for a call to an underlying function that does the real work; so you must first create the underlying function before you can create the operator. However, an operator is not merely syntactic sugar, because it carries additional information that helps the query planner optimize queries that use the operator. The next section will be devoted to explaining that additional information.
+
+HAWQ supports left unary, right unary, and binary operators. Operators can be overloaded; that is, the same operator name can be used for different operators that have different numbers and types of operands. When a query is executed, the system determines the operator to call from the number and types of the provided operands.
+
+Here is an example of creating an operator for adding two complex numbers. We assume we've already created the definition of type complex. First we need a function that does the work, then we can define the operator:
+
+``` sql
+CREATE FUNCTION complex_add(complex, complex)
+ RETURNS complex
+ AS 'filename', 'complex_add'
+ LANGUAGE C IMMUTABLE STRICT;
+
+CREATE OPERATOR + (
+ leftarg = complex,
+ rightarg = complex,
+ procedure = complex_add,
+ commutator = +
+);
+```
+
+Now we could execute a query like this:
+
+``` sql
+SELECT (a + b) AS c FROM test_complex;
+```
+
+```
+ c
+-----------------
+ (5.2,6.05)
+ (133.42,144.95)
+```
+
+We've shown how to create a binary operator here. To create unary operators, just omit one of leftarg (for left unary) or rightarg (for right unary). The procedure clause and the argument clauses are the only required items in CREATE OPERATOR. The commutator clause shown in the example is an optional hint to the query optimizer. Further details aboutcommutator and other optimizer hints appear in the next section.
+
+## Built-in Functions and Operators<a id="topic29"></a>
+
+The following table lists the categories of built-in functions and operators supported by PostgreSQL. All functions and operators are supported in HAWQ as in PostgreSQL with the exception of `STABLE` and `VOLATILE` functions, which are subject to the restrictions noted in [Using Functions in HAWQ](#topic27). See the [Functions and Operators](http://www.postgresql.org/docs/8.2/static/functions.html) section of the PostgreSQL documentation for more information about these built-in functions and operators.
+
+<a id="topic29__in204913"></a>
+
+<table>
+<caption><span class="tablecap">Table 2. Built-in functions and operators</span></caption>
+<colgroup>
+<col width="33%" />
+<col width="33%" />
+<col width="33%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Operator/Function Category</th>
+<th>VOLATILE Functions</th>
+<th>STABLE Functions</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td><a href="http://www.postgresql.org/docs/8.2/static/functions.html#FUNCTIONS-LOGICAL">Logical Operators</a></td>
+<td> </td>
+<td> </td>
+</tr>
+<tr class="even">
+<td><a href="http://www.postgresql.org/docs/8.2/static/functions-comparison.html">Comparison Operators</a></td>
+<td> </td>
+<td> </td>
+</tr>
+<tr class="odd">
+<td><a href="http://www.postgresql.org/docs/8.2/static/functions-math.html">Mathematical Functions and Operators</a></td>
+<td>random
+<p>setseed</p></td>
+<td> </td>
+</tr>
+<tr class="even">
+<td><a href="http://www.postgresql.org/docs/8.2/static/functions-string.html">String Functions and Operators</a></td>
+<td><em>All built-in conversion functions</em></td>
+<td>convert
+<p>pg_client_encoding</p></td>
+</tr>
+<tr class="odd">
+<td><a href="http://www.postgresql.org/docs/8.2/static/functions-binarystring.html">Binary String Functions and Operators</a></td>
+<td> </td>
+<td> </td>
+</tr>
+<tr class="even">
+<td><a href="http://www.postgresql.org/docs/8.2/static/functions-bitstring.html">Bit String Functions and Operators</a></td>
+<td> </td>
+<td> </td>
+</tr>
+<tr class="odd">
+<td><a href="http://www.postgresql.org/docs/8.3/static/functions-matching.html">Pattern Matching</a></td>
+<td> </td>
+<td> </td>
+</tr>
+<tr class="even">
+<td><a href="http://www.postgresql.org/docs/8.2/static/functions-formatting.html">Data Type Formatting Functions</a></td>
+<td> </td>
+<td>to_char
+<p>to_timestamp</p></td>
+</tr>
+<tr class="odd">
+<td><a href="http://www.postgresql.org/docs/8.2/static/functions-datetime.html">Date/Time Functions and Operators</a></td>
+<td>timeofday</td>
+<td>age
+<p>current_date</p>
+<p>current_time</p>
+<p>current_timestamp</p>
+<p>localtime</p>
+<p>localtimestamp</p>
+<p>now</p></td>
+</tr>
+<tr class="even">
+<td><a href="http://www.postgresql.org/docs/8.2/static/functions-geometry.html">Geometric Functions and Operators</a></td>
+<td> </td>
+<td> </td>
+</tr>
+<tr class="odd">
+<td><a href="http://www.postgresql.org/docs/8.2/static/functions-net.html">Network Address Functions and Operators</a></td>
+<td> </td>
+<td> </td>
+</tr>
+<tr class="even">
+<td><a href="http://www.postgresql.org/docs/8.2/static/functions-sequence.html">Sequence Manipulation Functions</a></td>
+<td>currval
+<p>lastval</p>
+<p>nextval</p>
+<p>setval</p></td>
+<td> </td>
+</tr>
+<tr class="odd">
+<td><a href="http://www.postgresql.org/docs/8.2/static/functions-conditional.html">Conditional Expressions</a></td>
+<td> </td>
+<td> </td>
+</tr>
+<tr class="even">
+<td><a href="http://www.postgresql.org/docs/8.2/static/functions-array.html">Array Functions and Operators</a></td>
+<td> </td>
+<td><em>All array functions</em></td>
+</tr>
+<tr class="odd">
+<td><a href="http://www.postgresql.org/docs/8.2/static/functions-aggregate.html">Aggregate Functions</a></td>
+<td> </td>
+<td> </td>
+</tr>
+<tr class="even">
+<td><a href="http://www.postgresql.org/docs/8.2/static/functions-subquery.html">Subquery Expressions</a></td>
+<td> </td>
+<td> </td>
+</tr>
+<tr class="odd">
+<td><a href="http://www.postgresql.org/docs/8.2/static/functions-comparisons.html">Row and Array Comparisons</a></td>
+<td> </td>
+<td> </td>
+</tr>
+<tr class="even">
+<td><a href="http://www.postgresql.org/docs/8.2/static/functions-srf.html">Set Returning Functions</a></td>
+<td>generate_series</td>
+<td> </td>
+</tr>
+<tr class="odd">
+<td><a href="http://www.postgresql.org/docs/8.2/static/functions-info.html">System Information Functions</a></td>
+<td> </td>
+<td><em>All session information functions</em>
+<p><em>All access privilege inquiry functions</em></p>
+<p><em>All schema visibility inquiry functions</em></p>
+<p><em>All system catalog information functions</em></p>
+<p><em>All comment information functions</em></p></td>
+</tr>
+<tr class="even">
+<td><a href="http://www.postgresql.org/docs/8.2/static/functions-admin.html">System Administration Functions</a></td>
+<td>set_config
+<p>pg_cancel_backend</p>
+<p>pg_reload_conf</p>
+<p>pg_rotate_logfile</p>
+<p>pg_start_backup</p>
+<p>pg_stop_backup</p>
+<p>pg_size_pretty</p>
+<p>pg_ls_dir</p>
+<p>pg_read_file</p>
+<p>pg_stat_file</p></td>
+<td>current_setting
+<p><em>All database object size functions</em></p></td>
+</tr>
+<tr class="odd">
+<td><a href="http://www.postgresql.org/docs/9.1/interactive/functions-xml.html">XML Functions</a></td>
+<td> </td>
+<td>xmlagg(xml)
+<p>xmlexists(text, xml)</p>
+<p>xml_is_well_formed(text)</p>
+<p>xml_is_well_formed_document(text)</p>
+<p>xml_is_well_formed_content(text)</p>
+<p>xpath(text, xml)</p>
+<p>xpath(text, xml, text[])</p>
+<p>xpath_exists(text, xml)</p>
+<p>xpath_exists(text, xml, text[])</p>
+<p>xml(text)</p>
+<p>text(xml)</p>
+<p>xmlcomment(xml)</p>
+<p>xmlconcat2(xml, xml)</p></td>
+</tr>
+</tbody>
+</table>
+
+## Window Functions<a id="topic30"></a>
+
+The following built-in window functions are HAWQ extensions to the PostgreSQL database. All window functions are *immutable*. For more information about window functions, see [Window Expressions](defining-queries.html#topic13).
+
+<a id="topic30__in164369"></a>
+
+<span class="tablecap">Table 3. Window functions</span>
+
+| Function | Return Type | Full Syntax | Description |
+|------------------------------------------------------|---------------------------|-----------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `cume_dist()` | `double precision` | `CUME_DIST() OVER ( [PARTITION BY ` *expr* `] ORDER BY ` *expr* ` )` | Calculates the cumulative distribution of a value in a group of values. Rows with equal values always evaluate to the same cumulative distribution value. |
+| `dense_rank()` | `bigint` | `DENSE_RANK () OVER ( [PARTITION BY ` *expr* `] ORDER BY ` *expr* `)` | Computes the rank of a row in an ordered group of rows without skipping rank values. Rows with equal values are given the same rank value. |
+| `first_value(expr)` | same as input *expr* type | FIRST\_VALUE expr ) OVER ( \[PARTITION BY expr \] ORDER BY expr \[ROWS|RANGE frame\_expr \] ) | Returns the first value in an ordered set of values. |
+| `lag(expr [,offset] [,default])` | same as input *expr* type | `LAG(` *expr* ` [,` *offset* `] [,` *default* `]) OVER ( [PARTITION BY ` *expr* `] ORDER BY ` *expr* ` )` | Provides access to more than one row of the same table without doing a self join. Given a series of rows returned from a query and a position of the cursor, `LAG` provides access to a row at a given physical offset prior to that position. The default `offset` is 1. *default* sets the value that is returned if the offset goes beyond the scope of the window. If *default* is not specified, the default value is null. |
+| `last_valueexpr` | same as input *expr* type | LAST\_VALUE(expr) OVER ( \[PARTITION BY expr\] ORDER BY expr \[ROWS|RANGE frame\_expr\] ) | Returns the last value in an ordered set of values. |
+| ` lead(expr [,offset] [,default])` | same as input *expr* type | `LEAD(expr [,offset] [,exprdefault]) OVER ( [PARTITION BY expr] ORDER BY expr )` | Provides access to more than one row of the same table without doing a self join. Given a series of rows returned from a query and a position of the cursor, `lead` provides access to a row at a given physical offset after that position. If *offset* is not specified, the default offset is 1. *default* sets the value that is returned if the offset goes beyond the scope of the window. If *default* is not specified, the default value is null. |
+| `ntile(expr)` | bigint | `NTILE(expr) OVER ( [PARTITION BY expr] ORDER BY expr )` | Divides an ordered data set into a number of buckets (as defined by *expr*) and assigns a bucket number to each row. |
+| `percent_rank(`) | `double precision` | `PERCENT_RANK () OVER ( [PARTITION BY expr] ORDER BY expr )` | Calculates the rank of a hypothetical row `R` minus 1, divided by 1 less than the number of rows being evaluated (within a window partition). |
+| `rank()` | bigint | `RANK () OVER ( [PARTITION BY expr] ORDER BY expr )` | Calculates the rank of a row in an ordered group of values. Rows with equal values for the ranking criteria receive the same rank. The number of tied rows are added to the rank number to calculate the next rank value. Ranks may not be consecutive numbers in this case. |
+| `row_number(`) | `bigint` | `ROW_NUMBER () OVER ( [PARTITION BY expr] ORDER BY expr )` | Assigns a unique number to each row to which it is applied (either each row in a window partition or each row of the query). |
+
+
+## Advanced Aggregate Functions<a id="topic31"></a>
+
+The following built-in advanced aggregate functions are HAWQ extensions of the PostgreSQL database.
+
+<a id="topic31__in2073121"></a>
+
+<table>
+
+<caption><span class="tablecap">Table 4. Advanced Aggregate Functions</span></caption>
+<colgroup>
+<col width="25%" />
+<col width="25%" />
+<col width="25%" />
+<col width="25%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Function</th>
+<th>Return Type</th>
+<th>Full Syntax</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td><code class="ph codeph">MEDIAN (expr)</code></td>
+<td><code class="ph codeph">timestamp, timestampz, interval, float</code></td>
+<td><code class="ph codeph">MEDIAN (expression)</code>
+<p><em>Example:</em></p>
+<pre class="pre codeblock"><code>SELECT department_id, MEDIAN(salary)
+FROM employees
+GROUP BY department_id; </code></pre></td>
+<td>Can take a two-dimensional array as input. Treats such arrays as matrices.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">PERCENTILE_CONT (expr) WITHIN GROUP (ORDER BY expr [DESC/ASC])</code></td>
+<td><code class="ph codeph">timestamp, timestampz, interval, float</code></td>
+<td><code class="ph codeph">PERCENTILE_CONT(percentage) WITHIN GROUP (ORDER BY expression)</code>
+<p><em>Example:</em></p>
+<pre class="pre codeblock"><code>SELECT department_id,
+PERCENTILE_CONT (0.5) WITHIN GROUP (ORDER BY salary DESC)
+"Median_cont";
+FROM employees GROUP BY department_id;</code></pre></td>
+<td>Performs an inverse function that assumes a continuous distribution model. It takes a percentile value and a sort specification and returns the same datatype as the numeric datatype of the argument. This returned value is a computed result after performing linear interpolation. Null are ignored in this calculation.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">PERCENTILE_DISC (expr) WITHIN GROUP (ORDER BY expr [DESC/ASC]</code>)</td>
+<td><code class="ph codeph">timestamp, timestampz, interval, float</code></td>
+<td><code class="ph codeph">PERCENTILE_DISC(percentage) WITHIN GROUP (ORDER BY expression)</code>
+<p><em>Example:</em></p>
+<pre class="pre codeblock"><code>SELECT department_id,
+PERCENTILE_DISC (0.5) WITHIN GROUP (ORDER BY salary DESC)
+"Median_desc";
+FROM employees GROUP BY department_id;</code></pre></td>
+<td>Performs an inverse distribution function that assumes a discrete distribution model. It takes a percentile value and a sort specification. This returned value is an element from the set. Null are ignored in this calculation.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">sum(array[])</code></td>
+<td><code class="ph codeph">smallint[]int[], bigint[], float[]</code></td>
+<td><code class="ph codeph">sum(array[[1,2],[3,4]])</code>
+<p><em>Example:</em></p>
+<pre class="pre codeblock"><code>CREATE TABLE mymatrix (myvalue int[]);
+INSERT INTO mymatrix VALUES (array[[1,2],[3,4]]);
+INSERT INTO mymatrix VALUES (array[[0,1],[1,0]]);
+SELECT sum(myvalue) FROM mymatrix;
+ sum
+---------------
+ {{1,3},{4,4}}</code></pre></td>
+<td>Performs matrix summation. Can take as input a two-dimensional array that is treated as a matrix.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">pivot_sum (label[], label, expr)</code></td>
+<td><code class="ph codeph">int[], bigint[], float[]</code></td>
+<td><code class="ph codeph">pivot_sum( array['A1','A2'], attr, value)</code></td>
+<td>A pivot aggregation using sum to resolve duplicate entries.</td>
+</tr>
+</tbody>
+</table>
+
+
diff --git a/query/gporca/query-gporca-changed.html.md.erb b/query/gporca/query-gporca-changed.html.md.erb
new file mode 100644
index 0000000..041aa4b
--- /dev/null
+++ b/query/gporca/query-gporca-changed.html.md.erb
@@ -0,0 +1,17 @@
+---
+title: Changed Behavior with GPORCA
+---
+
+<span class="shortdesc">When GPORCA is enabled, HAWQ's behavior changes. This topic describes these changes.</span>
+
+- The command `CREATE TABLE AS` distributes table data randomly if the `DISTRIBUTED BY` clause is not specified and no primary or unique keys are specified.
+- Statistics are required on the root table of a partitioned table. The `ANALYZE` command generates statistics on both root and individual partition tables (leaf child tables). See the `ROOTPARTITION` clause for `ANALYZE` command.
+- Additional Result nodes in the query plan:
+ - Query plan `Assert` operator.
+ - Query plan `Partition selector` operator.
+ - Query plan `Split` operator.
+- When running `EXPLAIN`, the query plan generated by GPORCA is different than the plan generated by the legacy query optimizer.
+- HAWQ adds the log file message `Planner produced plan` when GPORCA is enabled and HAWQ falls back to the legacy query optimizer to generate the query plan.
+- HAWQ issues a warning when statistics are missing from one or more table columns. When executing an SQL command with GPORCA, HAWQ issues a warning if the command performance could be improved by collecting statistics on a column or set of columns referenced by the command. The warning is issued on the command line and information is added to the HAWQ log file. For information about collecting statistics on table columns, see the `ANALYZE` command.
+
+
diff --git a/query/gporca/query-gporca-enable.html.md.erb b/query/gporca/query-gporca-enable.html.md.erb
new file mode 100644
index 0000000..37ef916
--- /dev/null
+++ b/query/gporca/query-gporca-enable.html.md.erb
@@ -0,0 +1,71 @@
+---
+title: Enabling GPORCA
+---
+
+<span class="shortdesc">Precompiled versions of HAWQ that include the GPORCA query optimizer enable it by default, no additional configuration is required. To use the GPORCA query optimizer in a HAWQ built from source, your build must include GPORCA. You must also enable specific HAWQ server configuration parameters at or after install time: </span>
+
+- [Set the <code class="ph codeph">optimizer\_analyze\_root\_partition</code> parameter to <code class="ph codeph">on</code>](#topic_r5d_hv1_kr) to enable statistics collection for the root partition of a partitioned table.
+- Set the `optimizer` parameter to `on` to enable GPORCA. You can set the parameter at these levels:
+ - [A HAWQ system](#topic_byp_lqk_br)
+ - [A specific HAWQ database](#topic_pzr_3db_3r)
+ - [A session or query](#topic_lx4_vqk_br)
+
+**Important:** If you intend to execute queries on partitioned tables with GPORCA enabled, you must collect statistics on the partitioned table root partition with the `ANALYZE ROOTPARTITION` command. The command `ANALYZE ROOTPARTITION` collects statistics on the root partition of a partitioned table without collecting statistics on the leaf partitions. If you specify a list of column names for a partitioned table, the statistics for the columns and the root partition are collected. For information on the `ANALYZE` command, see [ANALYZE](../../reference/sql/ANALYZE.html).
+
+You can also use the HAWQ utility `analyzedb` to update table statistics. The HAWQ utility `analyzedb` can update statistics for multiple tables in parallel. The utility can also check table statistics and update statistics only if the statistics are not current or do not exist. For information about the `analyzedb` utility, see [analyzedb](../../reference/cli/admin_utilities/analyzedb.html#topic1).
+
+As part of routine database maintenance, you should refresh statistics on the root partition when there are significant changes to child leaf partition data.
+
+## Setting the optimizer\_analyze\_root\_partition Parameter<a id="topic_r5d_hv1_kr"></a>
+
+When the configuration parameter `optimizer_analyze_root_partition` is set to `on`, root partition statistics will be collected when `ANALYZE` is run on a partitioned table. Root partition statistics are required by GPORCA.
+
+1. Log into the HAWQ master host as `gpadmin`, the HAWQ administrator.
+2. Set the values of the server configuration parameters. These HAWQ `hawq config` utility commands sets the value of the parameters to `on`:
+
+ ``shell
+ $ hawq config -c optimizer_analyze_root_partition -v on
+ ```
+
+3. Reload the modified configuration in HAWQ. This HAWQ `hawq stop` utility command reloads the `hawq-site.xml` file of the master without shutting down HAWQ.
+
+ ``` shell
+ $ hawq stop master -u
+ ```
+
+## Enabling GPORCA for a System<a id="topic_byp_lqk_br"></a>
+
+Set the server configuration parameter `optimizer` for the HAWQ system.
+
+1. Log into the HAWQ master host as `gpadmin`, the HAWQ administrator.
+2. Set the value of the server configuration `optimizer` parameter. This HAWQ `hawq config` utility command sets the value to `on`:
+
+ ``` shell
+ $ hawq config -c optimizer -v on
+ ```
+
+3. Reload the modified configuration in HAWQ. This HAWQ `hawq stop` utility command reloads the `hawq-site.xml` file of the master without shutting down HAWQ.
+
+ ``` shell
+ $ hawq stop master -u
+ ```
+
+## Enabling GPORCA for a Database<a id="topic_pzr_3db_3r"></a>
+
+Set the server configuration parameter `optimizer` for individual HAWQ databases with the `ALTER DATABASE` command. For example, this command enables GPORCA for the database *test\_db*.
+
+``` sql
+> ALTER DATABASE test_db SET OPTIMIZER = ON ;
+```
+
+## Enabling GPORCA for a Session or a Query<a id="topic_lx4_vqk_br"></a>
+
+You can use the `SET` command to set `optimizer` server configuration parameter for a session. For example, after you use the `psql` utility to connect to HAWQ, this `SET` command enables GPORCA:
+
+``` sql
+> set optimizer = on ;
+```
+
+To set the parameter for a specific query, include the `SET` command prior to running the query.
+
+
diff --git a/query/gporca/query-gporca-fallback.html.md.erb b/query/gporca/query-gporca-fallback.html.md.erb
new file mode 100644
index 0000000..14aeb2d
--- /dev/null
+++ b/query/gporca/query-gporca-fallback.html.md.erb
@@ -0,0 +1,142 @@
+---
+title: Determining The Query Optimizer In Use
+---
+
+<span class="shortdesc"> When GPORCA is enabled, you can determine if HAWQ is using GPORCA or is falling back to the legacy query optimizer. </span>
+
+These are two ways to determine which query optimizer HAWQ used to execute the query:
+
+- Examine `EXPLAIN` query plan output for the query. (Your output may include other settings.)
+ - When GPORCA generates the query plan, the GPORCA version is displayed near the end of the query plan . For example.
+
+ ``` pre
+ Settings: optimizer=on
+ Optimizer status: PQO version 1.627
+ ```
+
+ When HAWQ falls back to the legacy optimizer to generate the plan, `legacy query optimizer` is displayed near the end of the query plan. For example.
+
+ ``` pre
+ Settings: optimizer=on
+ Optimizer status: legacy query optimizer
+ ```
+
+ When the server configuration parameter `OPTIMIZER` is `off`, the following lines are displayed near the end of a query plan.
+
+ ``` pre
+ Settings: optimizer=off
+ Optimizer status: legacy query optimizer
+ ```
+
+ - These plan items appear only in the `EXPLAIN` plan output generated by GPORCA. The items are not supported in a legacy optimizer query plan.
+ - Assert operator
+ - Sequence operator
+ - DynamicIndexScan
+ - DynamicTableScan
+ - Table Scan
+ - When a query against a partitioned table is generated by GPORCA, the `EXPLAIN` plan displays only the number of partitions that are being eliminated is listed. The scanned partitions are not shown. The `EXPLAIN` plan generated by the legacy optimizer lists the scanned partitions.
+
+- View the log messages in the HAWQ log file.
+
+ The log file contains messages that indicate which query optimizer was used. In the log file message, the `[OPT]` flag appears when GPORCA attempts to optimize a query. If HAWQ falls back to the legacy optimizer, an error message is added to the log file, indicating the unsupported feature. Also, in the message, the label `Planner produced plan:` appears before the query when HAWQ falls back to the legacy optimizer.
+
+ **Note:** You can configure HAWQ to display log messages on the psql command line by setting the HAWQ server configuration parameter `client_min_messages` to `LOG`. See [Server Configuration Parameter Reference](../../reference/HAWQSiteConfig.html) for information about the parameter.
+
+## Example<a id="topic_n4w_nb5_xr"></a>
+
+This example shows the differences for a query that is run against partitioned tables when GPORCA is enabled.
+
+This `CREATE TABLE` statement creates a table with single level partitions:
+
+``` sql
+CREATE TABLE sales (trans_id int, date date,
+ amount decimal(9,2), region text)
+ DISTRIBUTED BY (trans_id)
+ PARTITION BY RANGE (date)
+ (START (date '20110101')
+ INCLUSIVE END (date '20120101')
+ EXCLUSIVE EVERY (INTERVAL '1 month'),
+ DEFAULT PARTITION outlying_dates);
+```
+
+This query against the table is supported by GPORCA and does not generate errors in the log file:
+
+``` sql
+select * from sales;
+```
+
+The `EXPLAIN` plan output lists only the number of selected partitions.
+
+```
+ -> Partition Selector for sales (dynamic scan id: 1) (cost=10.00..100.00 rows=50 width=4)
+ Partitions selected: 13 (out of 13)
+```
+
+Output from the log file indicates that GPORCA attempted to optimize the query:
+
+```
+2015-05-06 15:00:53.293451 PDT,"gpadmin","test",p2809,th297883424,"[local]",
+ ,2015-05-06 14:59:21 PDT,1120,con6,cmd1,seg-1,,dx3,x1120,sx1,"LOG","00000"
+ ,"statement: explain select * from sales
+;",,,,,,"explain select * from sales
+;",0,,"postgres.c",1566,
+
+2015-05-06 15:00:54.258412 PDT,"gpadmin","test",p2809,th297883424,"[local]",
+ ,2015-05-06 14:59:21 PDT,1120,con6,cmd1,seg-1,,dx3,x1120,sx1,"LOG","00000","
+[OPT]: Using default search strategy",,,,,,"explain select * from sales
+;",0,,"COptTasks.cpp",677,
+```
+
+The following cube query is not supported by GPORCA.
+
+``` sql
+select count(*) from foo group by cube(a,b);
+```
+
+The following EXPLAIN plan output includes the message "Feature not supported by GPORCA."
+
+``` sql
+postgres=# explain select count(*) from foo group by cube(a,b);
+```
+```
+LOG: statement: explain select count(*) from foo group by cube(a,b);
+LOG: 2016-04-14 16:26:15:487935 PDT,THD000,NOTICE,"Feature not supported by the GPORCA: Cube",
+LOG: Planner produced plan :0
+ QUERY PLAN
+---------------------------------------------------------------------------------------------------------------------------
+ Gather Motion 3:1 (slice3; segments: 3) (cost=9643.62..19400.26 rows=40897 width=28)
+ -> Append (cost=9643.62..19400.26 rows=13633 width=28)
+ -> HashAggregate (cost=9643.62..9993.39 rows=9328 width=28)
+ Group By: "rollup".unnamed_attr_2, "rollup".unnamed_attr_1, "rollup"."grouping", "rollup"."group_id"
+ -> Subquery Scan "rollup" (cost=8018.50..9589.81 rows=1435 width=28)
+ -> Redistribute Motion 3:3 (slice1; segments: 3) (cost=8018.50..9546.76 rows=1435 width=28)
+ Hash Key: "rollup".unnamed_attr_2, "rollup".unnamed_attr_1, "grouping", group_id()
+ -> GroupAggregate (cost=8018.50..9460.66 rows=1435 width=28)
+ Group By: "rollup"."grouping", "rollup"."group_id"
+ -> Subquery Scan "rollup" (cost=8018.50..9326.13 rows=2153 width=28)
+ -> GroupAggregate (cost=8018.50..9261.56 rows=2153 width=28)
+ Group By: "rollup".unnamed_attr_2, "rollup"."grouping", "rollup"."group_id"
+ -> Subquery Scan "rollup" (cost=8018.50..9073.22 rows=2870 width=28)
+ -> GroupAggregate (cost=8018.50..8987.12 rows=2870 width=28)
+ Group By: public.foo.b, public.foo.a
+ -> Sort (cost=8018.50..8233.75 rows=28700 width=8)
+ Sort Key: public.foo.b, public.foo.a
+ -> Seq Scan on foo (cost=0.00..961.00 rows=28700 width=8)
+ -> HashAggregate (cost=9116.27..9277.71 rows=4305 width=28)
+ Group By: "rollup".unnamed_attr_1, "rollup".unnamed_attr_2, "rollup"."grouping", "rollup"."group_id"
+ -> Subquery Scan "rollup" (cost=8018.50..9062.46 rows=1435 width=28)
+ -> Redistribute Motion 3:3 (slice2; segments: 3) (cost=8018.50..9019.41 rows=1435 width=28)
+ Hash Key: public.foo.a, public.foo.b, "grouping", group_id()
+ -> GroupAggregate (cost=8018.50..8933.31 rows=1435 width=28)
+ Group By: public.foo.a
+ -> Sort (cost=8018.50..8233.75 rows=28700 width=8)
+ Sort Key: public.foo.a
+ -> Seq Scan on foo (cost=0.00..961.00 rows=28700 width=8)
+ Settings: optimizer=on
+ Optimizer status: legacy query optimizer
+(30 rows)
+```
+
+Since this query is not supported by GPORCA, HAWQ falls back to the legacy optimizer.
+
+
diff --git a/query/gporca/query-gporca-features.html.md.erb b/query/gporca/query-gporca-features.html.md.erb
new file mode 100644
index 0000000..13eb04b
--- /dev/null
+++ b/query/gporca/query-gporca-features.html.md.erb
@@ -0,0 +1,215 @@
+---
+title: GPORCA Features and Enhancements
+---
+
+GPORCA includes enhancements for specific types of queries and operations. GPORCA also includes these optimization enhancements:
+
+- Improved join ordering
+- Join-Aggregate reordering
+- Sort order optimization
+- Data skew estimates included in query optimization
+
+## Queries Against Partitioned Tables<a id="topic_dwy_zml_gr"></a>
+
+GPORCA includes these enhancements for queries against partitioned tables:
+
+- Partition elimination is improved.
+- Query plan can contain the `Partition selector` operator.
+- Partitions are not enumerated in `EXPLAIN` plans.
+
+ For queries that involve static partition selection where the partitioning key is compared to a constant, GPORCA lists the number of partitions to be scanned in the `EXPLAIN` output under the Partition Selector operator. This example Partition Selector operator shows the filter and number of partitions selected:
+
+ ``` pre
+ Partition Selector for Part_Table (dynamic scan id: 1)
+ Filter: a > 10
+ Partitions selected: 1 (out of 3)
+ ```
+
+ For queries that involve dynamic partition selection where the partitioning key is compared to a variable, the number of partitions that are scanned will be known only during query execution. The partitions selected are not shown in the `EXPLAIN` output.
+
+- Plan size is independent of number of partitions.
+- Out of memory errors caused by number of partitions are reduced.
+
+This example `CREATE TABLE` command creates a range partitioned table.
+
+``` sql
+CREATE TABLE sales(order_id int, item_id int, amount numeric(15,2),
+ date date, yr_qtr int)
+ range partitioned by yr_qtr;
+```
+
+GPORCA improves on these types of queries against partitioned tables:
+
+- Full table scan. Partitions are not enumerated in plans.
+
+ ``` sql
+ SELECT * FROM sales;
+ ```
+
+- Query with a constant filter predicate. Partition elimination is performed.
+
+ ``` sql
+ SELECT * FROM sales WHERE yr_qtr = 201201;
+ ```
+
+- Range selection. Partition elimination is performed.
+
+ ``` sql
+ SELECT * FROM sales WHERE yr_qtr BETWEEN 201301 AND 201404 ;
+ ```
+
+- Joins involving partitioned tables. In this example, the partitioned dimension table *date\_dim* is joined with fact table *catalog\_sales*:
+
+ ``` sql
+ SELECT * FROM catalog_sales
+ WHERE date_id IN (SELECT id FROM date_dim WHERE month=12);
+ ```
+
+## Queries that Contain Subqueries<a id="topic_vph_wml_gr"></a>
+
+GPORCA handles subqueries more efficiently. A subquery is query that is nested inside an outer query block. In the following query, the `SELECT` in the `WHERE` clause is a subquery.
+
+``` sql
+SELECT * FROM part
+ WHERE price > (SELECT avg(price) FROM part);
+```
+
+GPORCA also handles queries that contain a correlated subquery (CSQ) more efficiently. A correlated subquery is a subquery that uses values from the outer query. In the following query, the `price` column is used in both the outer query and the subquery.
+
+``` sql
+SELECT * FROM part p1
+ WHERE price > (SELECT avg(price) FROM part p2
+ WHERE p2.brand = p1.brand);
+```
+
+GPORCA generates more efficient plans for the following types of subqueries:
+
+- CSQ in the `SELECT` list.
+
+ ``` sql
+ SELECT *,
+ (SELECT min(price) FROM part p2 WHERE p1.brand = p2.brand)
+ AS foo
+ FROM part p1;
+ ```
+
+- CSQ in disjunctive (`OR`) filters.
+
+ ``` sql
+ SELECT FROM part p1 WHERE p_size > 40 OR
+ p_retailprice >
+ (SELECT avg(p_retailprice)
+ FROM part p2
+ WHERE p2.p_brand = p1.p_brand)
+ ```
+
+- Nested CSQ with skip level correlations
+
+ ``` sql
+ SELECT * FROM part p1 WHERE p1.p_partkey
+ IN (SELECT p_partkey FROM part p2 WHERE p2.p_retailprice =
+ (SELECT min(p_retailprice)
+ FROM part p3
+ WHERE p3.p_brand = p1.p_brand)
+ );
+ ```
+
+ **Note:** Nested CSQ with skip level correlations are not supported by the legacy query optimizer.
+
+- CSQ with aggregate and inequality. This example contains a CSQ with an inequality.
+
+ ``` sql
+ SELECT * FROM part p1 WHERE p1.p_retailprice =
+ (SELECT min(p_retailprice) FROM part p2 WHERE p2.p_brand <> p1.p_brand);
+ ```
+
+<!-- -->
+
+- CSQ that must return one row.
+
+ ``` sql
+ SELECT p_partkey,
+ (SELECT p_retailprice FROM part p2 WHERE p2.p_brand = p1.p_brand )
+ FROM part p1;
+ ```
+
+## Queries that Contain Common Table Expressions<a id="topic_c3v_rml_gr"></a>
+
+GPORCA handles queries that contain the `WITH` clause. The `WITH` clause, also known as a common table expression (CTE), generates temporary tables that exist only for the query. This example query contains a CTE.
+
+``` sql
+WITH v AS (SELECT a, sum(b) as s FROM T where c < 10 GROUP BY a)
+ SELECT *FROM v AS v1 , v AS v2
+ WHERE v1.a <> v2.a AND v1.s < v2.s;
+```
+
+As part of query optimization, GPORCA can push down predicates into a CTE. For example query, GPORCA pushes the equality predicates to the CTE.
+
+``` sql
+WITH v AS (SELECT a, sum(b) as s FROM T GROUP BY a)
+ SELECT *
+ FROM v as v1, v as v2, v as v3
+ WHERE v1.a < v2.a
+ AND v1.s < v3.s
+ AND v1.a = 10
+ AND v2.a = 20
+ AND v3.a = 30;
+```
+
+GPORCA can handle these types of CTEs:
+
+- CTE that defines one or multiple tables. In this query, the CTE defines two tables.
+
+ ``` sql
+ WITH cte1 AS (SELECT a, sum(b) as s FROM T
+ where c < 10 GROUP BY a),
+ cte2 AS (SELECT a, s FROM cte1 where s > 1000)
+ SELECT *
+ FROM cte1 as v1, cte2 as v2, cte2 as v3
+ WHERE v1.a < v2.a AND v1.s < v3.s;
+ ```
+
+- Nested CTEs.
+
+ ``` sql
+ WITH v AS (WITH w AS (SELECT a, b FROM foo
+ WHERE b < 5)
+ SELECT w1.a, w2.b
+ FROM w AS w1, w AS w2
+ WHERE w1.a = w2.a AND w1.a > 2)
+ SELECT v1.a, v2.a, v2.b
+ FROM v as v1, v as v2
+ WHERE v1.a < v2.a;
+ ```
+
+## DML Operation Enhancements with GPORCA<a id="topic_plx_mml_gr"></a>
+
+GPORCA contains enhancements for DML operations such as `INSERT`.
+
+- A DML node in a query plan is a query plan operator.
+ - Can appear anywhere in the plan, as a regular node (top slice only for now)
+ - Can have consumers
+- New query plan operator `Assert` is used for constraints checking.
+
+ This example plan shows the `Assert` operator.
+
+ ```
+ QUERY PLAN
+ ------------------------------------------------------------
+ Insert (cost=0.00..4.61 rows=3 width=8)
+ -> Assert (cost=0.00..3.37 rows=3 width=24)
+ Assert Cond: (dmlsource.a > 2) IS DISTINCT FROM
+ false
+ -> Assert (cost=0.00..2.25 rows=3 width=24)
+ Assert Cond: NOT dmlsource.b IS NULL
+ -> Result (cost=0.00..1.14 rows=3 width=24)
+ -> Table Scan on dmlsource
+ ```
+
+## Queries with Distinct Qualified Aggregates (DQA)<a id="topic_anl_t3t_pv"></a>
+
+GPORCA improves performance for queries that contain distinct qualified aggregates (DQA) without a grouping column and when the table is not distributed on the columns used by the DQA. When encountering these types of queries, GPORCA uses an alternative plan that evaluates the aggregate functions in three stages (local, intermediate, and global aggregations).
+
+See [optimizer\_prefer\_scalar\_dqa\_multistage\_agg](../../reference/guc/parameter_definitions.html#optimizer_prefer_scalar_dqa_multistage_agg) for information on the configuration parameter that controls this behavior.
+
+
diff --git a/query/gporca/query-gporca-limitations.html.md.erb b/query/gporca/query-gporca-limitations.html.md.erb
new file mode 100644
index 0000000..5fb9cfd
--- /dev/null
+++ b/query/gporca/query-gporca-limitations.html.md.erb
@@ -0,0 +1,37 @@
+---
+title: GPORCA Limitations
+---
+
+<span class="shortdesc">There are limitations in HAWQ when GPORCA is enabled. GPORCA and the legacy query optimizer currently coexist in HAWQ because GPORCA does not support all HAWQ features. </span>
+
+
+## Unsupported SQL Query Features<a id="topic_kgn_vxl_vp"></a>
+
+These HAWQ features are unsupported when GPORCA is enabled:
+
+- Indexed expressions
+- `PERCENTILE` window function
+- External parameters
+- SortMergeJoin (SMJ)
+- Ordered aggregations
+- These analytics extensions:
+ - CUBE
+ - Multiple grouping sets
+- These scalar operators:
+ - `ROW`
+ - `ROWCOMPARE`
+ - `FIELDSELECT`
+- Multiple `DISTINCT` qualified aggregate functions
+- Inverse distribution functions
+
+## Performance Regressions<a id="topic_u4t_vxl_vp"></a>
+
+When GPORCA is enabled in HAWQ, the following features are known performance regressions:
+
+- Short running queries - For GPORCA, short running queries might encounter additional overhead due to GPORCA enhancements for determining an optimal query execution plan.
+- `ANALYZE` - For GPORCA, the `ANALYZE` command generates root partition statistics for partitioned tables. For the legacy optimizer, these statistics are not generated.
+- DML operations - For GPORCA, DML enhancements including the support of updates on partition and distribution keys might require additional overhead.
+
+Also, enhanced functionality of the features from previous versions could result in additional time required when GPORCA executes SQL statements with the features.
+
+
diff --git a/query/gporca/query-gporca-notes.html.md.erb b/query/gporca/query-gporca-notes.html.md.erb
new file mode 100644
index 0000000..ed943e4
--- /dev/null
+++ b/query/gporca/query-gporca-notes.html.md.erb
@@ -0,0 +1,28 @@
+---
+title: Considerations when Using GPORCA
+---
+
+<span class="shortdesc"> To execute queries optimally with GPORCA, consider certain criteria for the query. </span>
+
+Ensure the following criteria are met:
+
+- The table does not contain multi-column partition keys.
+- The table does not contain multi-level partitioning.
+- The query does not run against master only tables such as the system table *pg\_attribute*.
+- Statistics have been collected on the root partition of a partitioned table.
+
+If the partitioned table contains more than 20,000 partitions, consider a redesign of the table schema.
+
+GPORCA generates minidumps to describe the optimization context for a given query. Use the minidump files to analyze HAWQ issues. The minidump file is located under the master data directory and uses the following naming format:
+
+`Minidump_date_time.mdp`
+
+For information about the minidump file, see the server configuration parameter `optimizer_minidump`.
+
+When the `EXPLAIN ANALYZE` command uses GPORCA, the `EXPLAIN` plan shows only the number of partitions that are being eliminated. The scanned partitions are not shown. To show name of the scanned partitions in the segment logs set the server configuration parameter `gp_log_dynamic_partition_pruning` to `on`. This example `SET` command enables the parameter.
+
+``` sql
+SET gp_log_dynamic_partition_pruning = on;
+```
+
+
diff --git a/query/gporca/query-gporca-optimizer.html.md.erb b/query/gporca/query-gporca-optimizer.html.md.erb
new file mode 100644
index 0000000..11814f8
--- /dev/null
+++ b/query/gporca/query-gporca-optimizer.html.md.erb
@@ -0,0 +1,39 @@
+---
+title: About GPORCA
+---
+
+In HAWQ, you can use GPORCA or the legacy query optimizer.
+
+**Note:** To use the GPORCA query optimizer, you must be running a version of HAWQ built with GPORCA, and GPORCA must be enabled in your HAWQ deployment.
+
+These sections describe GPORCA functionality and usage:
+
+- **[Overview of GPORCA](../../query/gporca/query-gporca-overview.html)**
+
+ GPORCA extends the planning and optimization capabilities of the HAWQ legacy optimizer.
+
+- **[GPORCA Features and Enhancements](../../query/gporca/query-gporca-features.html)**
+
+ GPORCA includes enhancements for specific types of queries and operations:
+
+- **[Enabling GPORCA](../../query/gporca/query-gporca-enable.html)**
+
+ Precompiled versions of HAWQ that include the GPORCA query optimizer enable it by default, no additional configuration is required. To use the GPORCA query optimizer in a HAWQ built from source, your build must include GPORCA. You must also enable specific HAWQ server configuration parameters at or after install time:
+
+- **[Considerations when Using GPORCA](../../query/gporca/query-gporca-notes.html)**
+
+ To execute queries optimally with GPORCA, consider certain criteria for the query.
+
+- **[Determining The Query Optimizer In Use](../../query/gporca/query-gporca-fallback.html)**
+
+ When GPORCA is enabled, you can determine if HAWQ is using GPORCA or is falling back to the legacy query optimizer.
+
+- **[Changed Behavior with GPORCA](../../query/gporca/query-gporca-changed.html)**
+
+ When GPORCA is enabled, HAWQ's behavior changes. This topic describes these changes.
+
+- **[GPORCA Limitations](../../query/gporca/query-gporca-limitations.html)**
+
+ There are limitations in HAWQ when GPORCA is enabled. GPORCA and the legacy query optimizer currently coexist in HAWQ because GPORCA does not support all HAWQ features.
+
+
diff --git a/query/gporca/query-gporca-overview.html.md.erb b/query/gporca/query-gporca-overview.html.md.erb
new file mode 100644
index 0000000..56f97eb
--- /dev/null
+++ b/query/gporca/query-gporca-overview.html.md.erb
@@ -0,0 +1,23 @@
+---
+title: Overview of GPORCA
+---
+
+<span class="shortdesc">GPORCA extends the planning and optimization capabilities of the HAWQ legacy optimizer. </span> GPORCA is extensible and achieves better optimization in multi-core architecture environments. When GPORCA is available in your HAWQ installation and enabled, HAWQ uses GPORCA to generate an execution plan for a query when possible.
+
+GPORCA also enhances HAWQ query performance tuning in the following areas:
+
+- Queries against partitioned tables
+- Queries that contain a common table expression (CTE)
+- Queries that contain subqueries
+
+The legacy and GPORCA query optimizers coexist in HAWQ. The default query optimizer is GPORCA. When GPORCA is available and enabled in your HAWQ installation, HAWQ uses GPORCA to generate an execution plan for a query when possible. If GPORCA cannot be used, the legacy query optimizer is used.
+
+The following flow chart shows how GPORCA fits into the query planning architecture:
+
+<img src="../../images/gporca.png" id="topic1__image_rf5_svc_fv" class="image" width="672" />
+
+You can inspect the log to determine whether GPORCA or the legacy query optimizer produced the plan. The log message, "Optimizer produced plan" indicates that GPORCA generated the plan for your query. If the legacy query optimizer generated the plan, the log message reads "Planner produced plan". See [Determining The Query Optimizer In Use](query-gporca-fallback.html#topic1).
+
+**Note:** All legacy query optimizer (planner) server configuration parameters are ignored by GPORCA. However, if HAWQ falls back to the legacy optimizer, the planner server configuration parameters will impact the query plan generation.
+
+
diff --git a/query/query-performance.html.md.erb b/query/query-performance.html.md.erb
new file mode 100644
index 0000000..77506ad
--- /dev/null
+++ b/query/query-performance.html.md.erb
@@ -0,0 +1,143 @@
+---
+title: Query Performance
+---
+
+<span class="shortdesc">HAWQ dynamically allocates resources to queries. Query performance depends on several factors such as data locality, number of virtual segments used for the query and general cluster health.</span>
+
+- Dynamic Partition Elimination
+
+ In HAWQ, values available only when a query runs are used to dynamically prune partitions, which improves query processing speed. Enable or disable dynamic partition elimination by setting the server configuration parameter `gp_dynamic_partition_pruning` to `ON` or `OFF`; it is `ON` by default.
+
+- Memory Optimizations
+
+ HAWQ allocates memory optimally for different operators in a query and frees and re-allocates memory during the stages of processing a query.
+
+- Runaway Query Termination
+
+ HAWQ can automatically terminate the most memory-intensive queries based on a memory usage threshold. The threshold is set as a configurable percentage ([runaway\_detector\_activation\_percent](../reference/guc/parameter_definitions.html#runaway_detector_activation_percent)) of the resource quota for the segment, which is calculated by HAWQ's resource manager.
+
+ If the amount of virtual memory utilized by a physical segment exceeds the calculated threshold, then HAWQ begins terminating queries based on memory usage, starting with the query that is consuming the largest amount of memory. Queries are terminated until the percentage of utilized virtual memory is below the specified percentage.
+
+ To calculate the memory usage threshold for runaway queries, HAWQ uses the following formula:
+
+ *vmem threshold* = (*virtual memory quota calculated by resource manager* + [hawq\_re\_memory\_overcommit\_max](../reference/guc/parameter_definitions.html#hawq_re_memory_overcommit_max)) \* [runaway\_detector\_activation\_percent](../reference/guc/parameter_definitions.html#runaway_detector_activation_percent).
+
+ For example, if HAWQ resource manager calculates a virtual memory quota of 9GB,` hawq_re_memory_overcommit_max` is set to 1GB and the value of `runaway_detector_activation_percent` is 95 (95%), then HAWQ starts terminating queries when the utilized virtual memory exceeds 9.5 GB.
+
+ To disable automatic query detection and termination, set the value of `runaway_detector_activation_percent` to 100.
+
+## How to Investigate Query Performance Issues<a id="id_xkg_znj_f5"></a>
+
+A query is not executing as quickly as you would expect. Here is how to investigate possible causes of slowdown:
+
+1. Check the health of the cluster.
+ 1. Are any DataNodes, segments or nodes down?
+ 2. Are there many failed disks?
+
+2. Check table statistics. Have the tables involved in the query been analyzed?
+3. Check the plan of the query and run /3/4 to determine the bottleneck.
+ Sometimes, there is not enough memory for some operators, such as Hash Join, or spill files are used. If an operator cannot perform all of its work in the memory allocated to it, it caches data on disk in *spill files*. Compared with no spill files, a query will run much slower.
+
+4. Check data locality statistics using /3/4. Alternately you can check the logs. Data locality result for every query could also be found in the log of HAWQ. See [Data Locality Statistics](query-performance.html#topic_amk_drc_d5) for information on the statistics.
+5. Check resource queue status. You can query view `pg_resqueue_status` to check if the target queue has already dispatched some resource to the queries, or if the target queue is lacking resources. See [Checking Existing Resource Queues](../resourcemgmt/ResourceQueues.html#topic_lqy_gls_zt).
+6. Analyze a dump of the resource manager's status to see more resource queue status. See [Analyzing Resource Manager Status](../resourcemgmt/ResourceQueues.html#topic_zrh_pkc_f5).
+
+## Data Locality Statistics<a id="topic_amk_drc_d5"></a>
+
+For visibility into query performance, use the EXPLAIN ANALYZE to obtain data locality statistics. For example:
+
+``` sql
+postgres=# create table test (i int);
+postgres=# insert into test values(2);
+postgres=# explain analyze select * from test;
+```
+```
+QUERY PLAN
+.......
+Data locality statistics:
+data locality ratio: 1.000; virtual segment number: 1; different host number: 1;
+virtual segment number per host(avg/min/max): (1/1/1);
+segment size(avg/min/max): (32.000 B/32 B/32 B);
+segment size with penalty(avg/min/max): (32.000 B/32 B/32 B);
+continuity(avg/min/max): (1.000/1.000/1.000); DFS metadatacache: 7.816 ms;
+resource allocation: 0.615 ms; datalocality calculation: 0.136 ms.
+```
+
+The following table describes the metrics related to data locality. Use these metrics to examine issues behind a query's performance.
+
+<a id="topic_amk_drc_d5__table_q4p_25c_d5"></a>
+
+<table>
+<caption><span class="tablecap">Table 1. Data Locality Statistics</span></caption>
+<colgroup>
+<col width="50%" />
+<col width="50%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Statistic</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>data locality ratio</td>
+<td><p>Indicates the total local read ratio of a query. The lower the ratio, the more remote read happens. Since remote read on HDFS needs network IO, the execution time of a query may increase.</p>
+<p>For hash distributed tables, all the blocks of a file will be processed by one segment, so if data on HDFS is redistributed, such as by the HDFS Balancer, the data locality ratio will be decreased. In this case, you can redistribute the hash distributed table manually by using CREATE TABLE AS SELECT.</p></td>
+</tr>
+<tr class="even">
+<td>number of virtual segments</td>
+<td>Typically, the more virtual segments are used, the faster the query will be executed. If the virtual segment number is too small, you can check whether <code class="ph codeph">default_hash_table_bucket_number</code>, <code class="ph codeph">hawq_rm_nvseg_perquery_limit</code>, or the bucket number of a hash distributed table is small. See <a href="#topic_wv3_gzc_d5">Number of Virtual Segments</a>.</td>
+</tr>
+<tr class="odd">
+<td>different host number</td>
+<td>Indicates how many hosts are used to run this query. All the hosts should be used when the virtual segment number is bigger than the total number of hosts according to the resource allocation strategy of HAWQ. As a result, if this metric is smaller than the total number of hosts for a big query, it often indicates that some hosts are down. In this case, use “select gp_segment_configuration” to check the node states first.</td>
+</tr>
+<tr class="even">
+<td>segment size and segment size with penalty</td>
+<td>“segment size” indicates the (avg/min/max) data size which is processed by a virtual segment. “segment size with penalty” is the segment size when remote read is calculated as “net_disk_ratio” * block size. The virtual segment that contains remote read should process less data than the virtual segment that contains only local read. “net_disk_ratio” can be tuned to measure how much slower the remote read is than local read for different network environments, while considering the workload balance between the nodes. The default value of “net_disk_ratio” is 1.01.</td>
+</tr>
+<tr class="odd">
+<td>continuity</td>
+<td>reading a HDFS file discontinuously will introduce additional seek, which will slow the table scan of a query. A low value of continuity indicates that the blocks of a file are not continuously distributed on a datanode.</td>
+</tr>
+<tr class="even">
+<td>DFS metadatacache</td>
+<td>Indicates the metadatacache time cost for a query. In HAWQ, HDFS block information is cached in a metadatacache process. If cache miss happens, time cost of metadatacache may increase.</td>
+</tr>
+<tr class="odd">
+<td>resource allocation</td>
+<td>Indicates the time cost of acquiring resources from the resource manager.</td>
+</tr>
+<tr class="even">
+<td>datalocality calculation</td>
+<td>Indicates the time to run the algorithm that assigns HDFS blocks to virtual segments and calculates the data locality ratio.</td>
+</tr>
+</tbody>
+</table>
+
+## Number of Virtual Segments<a id="topic_wv3_gzc_d5"></a>
+
+The number of virtual segment used has impacts on the query performance. HAWQ decides the number of virtual segments of a query (its parallelism) by using the following rules:
+
+- **Cost of the query**. Small queries use fewer segments and larger queries use more segments. Note that there are some techniques you can use when defining resource queues to influence the number of virtual segments and general resources that are allocated to queries. See [Best Practices for Using Resource Queues](../bestpractices/managing_resources_bestpractices.html#topic_hvd_pls_wv).
+- **Available resources**. Resources available at query time. If more resources are available in the resource queue, the resources will be used.
+- **Hash table and bucket number**. If the query involves only hash-distributed tables, and the bucket number (bucketnum) configured for all the hash tables is either the same bucket number for all tables or the table size for random tables is no more than 1.5 times larger than the size of hash tables for the hash tables, then the query's parallelism is fixed (equal to the hash table bucket number). Otherwise, the number of virtual segments depends on the query's cost and hash-distributed table queries will behave like queries on randomly distributed tables.
+- **Query Type**: For queries with some user-defined functions or for external tables where calculating resource costs is difficult , then the number of virtual segments is controlled by `hawq_rm_nvseg_perquery_limit `and `hawq_rm_nvseg_perquery_perseg_limit` parameters, as well as by the ON clause and the location list of external tables. If the query has a hash result table (e.g. `INSERT into hash_table`) then the number of virtual segment number must be equal to the bucket number of the resulting hash table, If the query is performed in utility mode, such as for `COPY` and `ANALYZE` operations, the virtual segment number is calculated by different policies, which will be explained later in this section.
+
+The following are guidelines for numbers of virtual segments to use, provided there are sufficient resources available.
+
+- **Random tables exist in the select list:** \#vseg (number of virtual segments) depends on the size of the table.
+- **Hash tables exist in the select list:** \#vseg depends on the bucket number of the table.
+- **Random and hash tables both exist in the select list:** \#vseg depends on the bucket number of the table, if the table size of random tables is no more than 1.5 times larger than the size of hash tables. Otherwise, \#vseg depends on the size of the random table.
+- **User-defined functions exist:** \#vseg depends on the `hawq_rm_nvseg_perquery_limit` and `hawq_rm_nvseg_perquery_perseg_limit` parameters.
+- **PXF external tables exist:** \#vseg depends on the `default_hash_table_bucket_number` parameter.
+- **gpfdist external tables exist:** \#vseg is at least the number of locations in the location list.
+- **The command for CREATE EXTERNAL TABLE is used:** \#vseg must reflect the value in the command and use the `ON` clause in the command.
+- **Hash tables are copied to or from files:** \#vseg depends on the bucket number of the hash table.
+- **Random tables are copied to files:** \#vseg depends on the size of the random table.
+- **Random tables are copied from files:** \#vseg is a fixed value. \#vseg is 6, when there are sufficient resources.
+- **ANALYZE table:** Analyzing a nonpartitioned table will use more virtual segments than a partitioned table.
+- **Relationship between hash distribution results:** \#vseg must be the same as the bucket number for the hash table.
+
+
diff --git a/query/query-profiling.html.md.erb b/query/query-profiling.html.md.erb
new file mode 100644
index 0000000..f711b84
--- /dev/null
+++ b/query/query-profiling.html.md.erb
@@ -0,0 +1,146 @@
+---
+title: Query Profiling
+---
+
+<span class="shortdesc">Examine the query plans of poorly performing queries to identify possible performance tuning opportunities.</span>
+
+HAWQ devises a *query plan* for each query. Choosing the right query plan to match the query and data structure is necessary for good performance. A query plan defines how HAWQ will run the query in the parallel execution environment.
+
+The query optimizer uses data statistics maintained by the database to choose a query plan with the lowest possible cost. Cost is measured in disk I/O, shown as units of disk page fetches. The goal is to minimize the total execution cost for the plan.
+
+View the plan for a given query with the `EXPLAIN` command. `EXPLAIN` shows the query optimizer's estimated cost for the query plan. For example:
+
+``` sql
+EXPLAIN SELECT * FROM names WHERE id=22;
+```
+
+`EXPLAIN ANALYZE` runs the statement in addition to displaying its plan. This is useful for determining how close the optimizer's estimates are to reality. For example:
+
+``` sql
+EXPLAIN ANALYZE SELECT * FROM names WHERE id=22;
+```
+
+**Note:** The legacy and GPORCA query optimizers coexist in HAWQ. GPORCA is the default HAWQ optimizer. HAWQ uses GPORCA to generate an execution plan for a query when possible. The `EXPLAIN` output generated by GPORCA is different than the output generated by the legacy query optimizer.
+
+When the `EXPLAIN ANALYZE` command uses GPORCA, the `EXPLAIN` plan shows only the number of partitions that are being eliminated. The scanned partitions are not shown. To show name of the scanned partitions in the segment logs set the server configuration parameter `gp_log_dynamic_partition_pruning` to `on`. This example `SET` command enables the parameter.
+
+``` sql
+SET gp_log_dynamic_partition_pruning = on;
+```
+
+For information about GPORCA, see [Querying Data](query.html#topic1).
+
+## Reading EXPLAIN Output<a id="topic40"></a>
+
+A query plan is a tree of nodes. Each node in the plan represents a single operation, such as a table scan, join, aggregation, or sort.
+
+Read plans from the bottom to the top: each node feeds rows into the node directly above it. The bottom nodes of a plan are usually table scan operations. If the query requires joins, aggregations, sorts, or other operations on the rows, there are additional nodes above the scan nodes to perform these operations. The topmost plan nodes are usually HAWQ motion nodes: redistribute, broadcast, or gather motions. These operations move rows between segment instances during query processing.
+
+The output of `EXPLAIN` has one line for each node in the plan tree and shows the basic node type and the following execution cost estimates for that plan node:
+
+- **cost** —Measured in units of disk page fetches. 1.0 equals one sequential disk page read. The first estimate is the start-up cost of getting the first row and the second is the total cost of cost of getting all rows. The total cost assumes all rows will be retrieved, which is not always true; for example, if the query uses `LIMIT`, not all rows are retrieved.
+- **rows** —The total number of rows output by this plan node. This number is usually less than the number of rows processed or scanned by the plan node, reflecting the estimated selectivity of any `WHERE` clause conditions. Ideally, the estimate for the topmost node approximates the number of rows that the query actually returns.
+- **width** —The total bytes of all the rows that this plan node outputs.
+
+Note the following:
+
+- The cost of a node includes the cost of its child nodes. The topmost plan node has the estimated total execution cost for the plan. This is the number the optimizer intends to minimize.
+- The cost reflects only the aspects of plan execution that the query optimizer takes into consideration. For example, the cost does not reflect time spent transmitting result rows to the client.
+
+### EXPLAIN Example<a id="topic41"></a>
+
+The following example describes how to read an `EXPLAIN` query plan for a query:
+
+``` sql
+EXPLAIN SELECT * FROM names WHERE name = 'Joelle';
+```
+
+```
+ QUERY PLAN
+-----------------------------------------------------------------------------
+ Gather Motion 2:1 (slice1; segments: 2) (cost=0.00..1.01 rows=1 width=11)
+ -> Append-only Scan on names (cost=0.00..1.01 rows=1 width=11)
+ Filter: name::text = 'Joelle'::text
+(3 rows)
+```
+
+Read the plan from the bottom to the top. To start, the query optimizer sequentially scans the *names* table. Notice the `WHERE` clause is applied as a *filter* condition. This means the scan operation checks the condition for each row it scans and outputs only the rows that satisfy the condition.
+
+The results of the scan operation are passed to a *gather motion* operation. In HAWQ, a gather motion is when segments send rows to the master. In this example, we have two segment instances that send to one master instance. This operation is working on `slice1` of the parallel query execution plan. A query plan is divided into *slices* so the segments can work on portions of the query plan in parallel.
+
+The estimated startup cost for this plan is `00.00` (no cost) and a total cost of `1.01` disk page fetches. The optimizer estimates this query will return one row.
+
+## Reading EXPLAIN ANALYZE Output<a id="topic42"></a>
+
+`EXPLAIN ANALYZE` plans and runs the statement. The `EXPLAIN ANALYZE` plan shows the actual execution cost along with the optimizer's estimates. This allows you to see if the optimizer's estimates are close to reality. `EXPLAIN ANALYZE` also shows the following:
+
+- The total runtime (in milliseconds) in which the query executed.
+- The memory used by each slice of the query plan, as well as the memory reserved for the whole query statement.
+- Statistics for the query dispatcher, including the number of executors used for the current query (total number/number of executors cached by previous queries/number of executors newly connected), dispatcher time (total dispatch time/connection establish time/dispatch data to executor time); and some time(max/min/avg) details for dispatching data, consuming executor data, and freeing executor.
+- Statistics about data locality. See [Data Locality Statistics](query-performance.html#topic_amk_drc_d5) for details about these statistics.
+- The number of *workers* (segments) involved in a plan node operation. Only segments that return rows are counted.
+- The Max/Last statistics are for the segment that output the maximum number of rows and the segment with the longest *<time> to end*.
+- The segment id of the segment that produced the most rows for an operation.
+- For relevant operations, the amount of memory (`work_mem`) used by the operation. If the `work_mem` was insufficient to perform the operation in memory, the plan shows the amount of data spilled to disk for the lowest-performing segment. For example:
+
+ ``` pre
+ Work_mem used: 64K bytes avg, 64K bytes max (seg0).
+ Work_mem wanted: 90K bytes avg, 90K byes max (seg0) to lessen
+ workfile I/O affecting 2 workers.
+ ```
+
+- The time (in milliseconds) in which the segment that produced the most rows retrieved the first row, and the time taken for that segment to retrieve all rows. The result may omit *<time> to first row* if it is the same as the *<time> to end*.
+
+### EXPLAIN ANALYZE Example<a id="topic43"></a>
+
+This example describes how to read an `EXPLAIN ANALYZE` query plan using the same query. The `bold` parts of the plan show actual timing and rows returned for each plan node, as well as memory and time statistics for the whole query.
+
+``` sql
+EXPLAIN ANALYZE SELECT * FROM names WHERE name = 'Joelle';
+```
+
+```
+ QUERY PLAN
+------------------------------------------------------------------------
+ Gather Motion 1:1 (slice1; segments: 1) (cost=0.00..1.01 rows=1 width=7)
+ Rows out: Avg 1.0 rows x 1 workers at destination. Max/Last(seg0:ip-10-0-1-16/seg0:ip-10-0-1-16) 1/1 rows with 8.713/8.713 ms to first row, 8.714/8.714 ms to end, start offset by 0.708/0.708 ms.
+ -> Append-only Scan on names (cost=0.00..1.01 rows=1 width=7)
+ Filter: name = 'Joelle'::text
+ Rows out: Avg 1.0 rows x 1 workers. Max/Last(seg0:ip-10-0-1-16/seg0:ip-10-0-1-16) 1/1 rows with 7.053/7.053 ms to first row, 7.089/7.089 ms to end, start offset by 2.162/2.162 ms.
+ Slice statistics:
+ (slice0) Executor memory: 159K bytes.
+ (slice1) Executor memory: 247K bytes (seg0:ip-10-0-1-16).
+ Statement statistics:
+ Memory used: 262144K bytes
+ Dispatcher statistics:
+ executors used(total/cached/new connection): (1/1/0); dispatcher time(total/connection/dispatch data): (0.217 ms/0.000 ms/0.037 ms).
+ dispatch data time(max/min/avg): (0.037 ms/0.037 ms/0.037 ms); consume executor data time(max/min/avg): (0.015 ms/0.015 ms/0.015 ms); free executor time(max/min/avg): (0.000 ms/0.000 ms/0.000 ms).
+ Data locality statistics:
+ data locality ratio: 1.000; virtual segment number: 1; different host number: 1; virtual segment number per host(avg/min/max): (1/1/1); segment size(avg/min/max): (48.000 B/48 B/48 B); segment size with penalty(avg/min/max): (48.000 B/48 B/48 B); continuity(avg/min/max): (1.000/1.000/1.000); DFS metadatacache: 9.343 ms; resource allocation: 0.638 ms; datalocality calculation: 0.144 ms.
+ Total runtime: 19.690 ms
+(16 rows)
+```
+
+Read the plan from the bottom to the top. The total elapsed time to run this query was *19.690* milliseconds.
+
+The *Append-only scan* operation had only one segment (*seg0*) that returned rows, and it returned just *1 row*. The Max/Last statistics are identical in this example because only one segment returned rows. It took *7.053* milliseconds to find the first row and *7.089* milliseconds to scan all rows. This result is close to the optimizer's estimate: the query optimizer estimated it would return one row for this query. The *gather motion* (segments sending data to the master) received 1 row. The total elapsed time for this operation was *19.690* milliseconds.
+
+## Examining Query Plans to Solve Problems<a id="topic44"></a>
+
+If a query performs poorly, examine its query plan and ask the following questions:
+
+- **Do operations in the plan take an exceptionally long time?** Look for an operation that consumes the majority of query processing time. For example, if a scan on a hash table takes longer than expected, the data locality may be low; reloading the data can increase the data locality and speed up the query. Or, adjust `enable_<operator>` parameters to see if you can force the legacy query optimizer (planner) to choose a different plan by disabling a particular query plan operator for that query.
+- **Are the optimizer's estimates close to reality?** Run `EXPLAIN ANALYZE` and see if the number of rows the optimizer estimates is close to the number of rows the query operation actually returns. If there is a large discrepancy, collect more statistics on the relevant columns.
+- **Are selective predicates applied early in the plan?** Apply the most selective filters early in the plan so fewer rows move up the plan tree. If the query plan does not correctly estimate query predicate selectivity, collect more statistics on the relevant columns. You can also try reordering the `WHERE` clause of your SQL statement.
+- **Does the optimizer choose the best join order?** When you have a query that joins multiple tables, make sure that the optimizer chooses the most selective join order. Joins that eliminate the largest number of rows should be done earlier in the plan so fewer rows move up the plan tree.
+
+ If the plan is not choosing the optimal join order, set `join_collapse_limit=1` and use explicit `JOIN` syntax in your SQL statement to force the legacy query optimizer (planner) to the specified join order. You can also collect more statistics on the relevant join columns.
+
+- **Does the optimizer selectively scan partitioned tables?** If you use table partitioning, is the optimizer selectively scanning only the child tables required to satisfy the query predicates? Scans of the parent tables should return 0 rows since the parent tables do not contain any data. See [Verifying Your Partition Strategy](../ddl/ddl-partition.html#topic74) for an example of a query plan that shows a selective partition scan.
+- **Does the optimizer choose hash aggregate and hash join operations where applicable?** Hash operations are typically much faster than other types of joins or aggregations. Row comparison and sorting is done in memory rather than reading/writing from disk. To enable the query optimizer to choose hash operations, there must be sufficient memory available to hold the estimated number of rows. Try increasing work memory to improve performance for a query. If possible, run an `EXPLAIN ANALYZE` for the query to show which plan operations spilled to disk, how much work memory they used, and how much memory was required to avoid spilling to disk. For example:
+
+ `Work_mem used: 23430K bytes avg, 23430K bytes max (seg0). Work_mem wanted: 33649K bytes avg, 33649K bytes max (seg0) to lessen workfile I/O affecting 2 workers.`
+
+ The "bytes wanted" message from `EXPLAIN ANALYZE` is based on the amount of data written to work files and is not exact. The minimum `work_mem` needed can differ from the suggested value.
+
+
diff --git a/query/query.html.md.erb b/query/query.html.md.erb
new file mode 100644
index 0000000..9c218c7
--- /dev/null
+++ b/query/query.html.md.erb
@@ -0,0 +1,37 @@
+---
+title: Querying Data
+---
+
+This topic provides information about using SQL in HAWQ databases.
+
+You enter SQL statements called queries to view and analyze data in a database using the `psql` interactive SQL client and other client tools.
+
+**Note:** HAWQ queries timeout after a period of 600 seconds. For this reason, long-running queries may appear to hang until results are processed or until the timeout period expires.
+
+- **[About HAWQ Query Processing](../query/HAWQQueryProcessing.html)**
+
+ This topic provides an overview of how HAWQ processes queries. Understanding this process can be useful when writing and tuning queries.
+
+- **[About GPORCA](../query/gporca/query-gporca-optimizer.html)**
+
+ In HAWQ, you can use GPORCA or the legacy query optimizer.
+
+- **[Defining Queries](../query/defining-queries.html)**
+
+ HAWQ is based on the PostgreSQL implementation of the SQL standard. SQL commands are typically entered using the standard PostgreSQL interactive terminal `psql`, but other programs that have similar functionality can be used as well.
+
+- **[Using Functions and Operators](../query/functions-operators.html)**
+
+ HAWQ evaluates functions and operators used in SQL expressions.
+
+- **[Query Performance](../query/query-performance.html)**
+
+ HAWQ dynamically allocates resources to queries. Query performance depends on several factors such as data locality, number of virtual segments used for the query and general cluster health.
+
+- **[Query Profiling](../query/query-profiling.html)**
+
+ Examine the query plans of poorly performing queries to identify possible performance tuning opportunities.
+
+
+
+
diff --git a/reference/CharacterSetSupportReference.html.md.erb b/reference/CharacterSetSupportReference.html.md.erb
new file mode 100644
index 0000000..c2a4754
--- /dev/null
+++ b/reference/CharacterSetSupportReference.html.md.erb
@@ -0,0 +1,439 @@
+---
+title: Character Set Support Reference
+---
+
+This topic provides a referene of the character sets supported in HAWQ.
+
+The character set support in HAWQ allows you to store text in a variety of character sets, including single-byte character sets such as the ISO 8859 series and multiple-byte character sets such as EUC (Extended Unix Code), UTF-8, and Mule internal code. All supported character sets can be used transparently by clients, but a few are not supported for use within the server (that is, as a server-side encoding). The default character set is selected while initializing your HAWQ using `hawq init.` It can be overridden when you create a database, so you can have multiple databases each with a different character set.
+
+<table style="width:100%;">
+<colgroup>
+<col width="16%" />
+<col width="16%" />
+<col width="16%" />
+<col width="16%" />
+<col width="16%" />
+<col width="16%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Name</th>
+<th>Description</th>
+<th>Language</th>
+<th>Server</th>
+<th>Bytes/Char</th>
+<th>Aliases</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>BIG5</td>
+<td>Big Five</td>
+<td>Traditional Chinese</td>
+<td>No</td>
+<td>1-2</td>
+<td>WIN950, Windows950</td>
+</tr>
+<tr class="even">
+<td>EUC_CN</td>
+<td>Extended UNIX Code-CN</td>
+<td>Simplified Chinese</td>
+<td>Yes</td>
+<td>1-3</td>
+<td></td>
+</tr>
+<tr class="odd">
+<td>EUC_JP</td>
+<td>Extended UNIX Code-JP</td>
+<td>Japanese</td>
+<td>Yes</td>
+<td>1-3</td>
+<td> </td>
+</tr>
+<tr class="even">
+<td>EUC_KR</td>
+<td>Extended UNIX Code-KR</td>
+<td>Korean</td>
+<td>Yes</td>
+<td>1-3</td>
+<td> </td>
+</tr>
+<tr class="odd">
+<td>EUC_TW</td>
+<td>Extended UNIX Code-TW</td>
+<td>Traditional Chinese, Taiwanese</td>
+<td>Yes</td>
+<td>1-3</td>
+<td> </td>
+</tr>
+<tr class="even">
+<td>GB18030</td>
+<td>National Standard</td>
+<td>Chinese</td>
+<td>No</td>
+<td>1-2</td>
+<td> </td>
+</tr>
+<tr class="odd">
+<td>GBK</td>
+<td>Extended National Standard</td>
+<td>Simplified Chinese</td>
+<td>No</td>
+<td>1-2</td>
+<td>WIN936,Windows936</td>
+</tr>
+<tr class="even">
+<td>ISO_8859_5</td>
+<td>ISO 8859-5, ECMA 113</td>
+<td>Latin/Cyrillic</td>
+<td>Yes</td>
+<td>1</td>
+<td> </td>
+</tr>
+<tr class="odd">
+<td>ISO_8859_6</td>
+<td>ISO 8859-6, ECMA 114</td>
+<td>Latin/Arabic</td>
+<td>Yes</td>
+<td>1</td>
+<td> </td>
+</tr>
+<tr class="even">
+<td>ISO_8859_7</td>
+<td>ISO 8859-7, ECMA 118</td>
+<td>Latin/Greek</td>
+<td>Yes</td>
+<td>1</td>
+<td> </td>
+</tr>
+<tr class="odd">
+<td>ISO_8859_8</td>
+<td>ISO 8859-8, ECMA 121</td>
+<td>Latin/Hebrew</td>
+<td>Yes</td>
+<td>1</td>
+<td> </td>
+</tr>
+<tr class="even">
+<td>JOHAB</td>
+<td>JOHA</td>
+<td>Korean (Hangul)</td>
+<td>Yes</td>
+<td>1-3</td>
+<td> </td>
+</tr>
+<tr class="odd">
+<td>KOI8</td>
+<td>KOI8-R(U)</td>
+<td>Cyrillic</td>
+<td>Yes</td>
+<td>1</td>
+<td>KOI8R</td>
+</tr>
+<tr class="even">
+<td>LATIN1</td>
+<td>ISO 8859-1, ECMA 94</td>
+<td>Western European</td>
+<td>Yes</td>
+<td>1</td>
+<td>ISO88591</td>
+</tr>
+<tr class="odd">
+<td>LATIN2</td>
+<td>ISO 8859-2, ECMA 94</td>
+<td>Central European</td>
+<td>Yes</td>
+<td>1</td>
+<td>ISO88592</td>
+</tr>
+<tr class="even">
+<td>LATIN3</td>
+<td>ISO 8859-3, ECMA 94</td>
+<td>South European</td>
+<td>Yes</td>
+<td>1</td>
+<td>ISO88593</td>
+</tr>
+<tr class="odd">
+<td>LATIN4</td>
+<td>ISO 8859-4, ECMA 94</td>
+<td>North European</td>
+<td>Yes</td>
+<td>1</td>
+<td>ISO88594</td>
+</tr>
+<tr class="even">
+<td>LATIN5</td>
+<td>ISO 8859-9, ECMA 128</td>
+<td>Turkish</td>
+<td>Yes</td>
+<td>1</td>
+<td>ISO88599</td>
+</tr>
+<tr class="odd">
+<td>LATIN6</td>
+<td>ISO 8859-10, ECMA 144</td>
+<td>Nordic</td>
+<td>Yes</td>
+<td>1</td>
+<td>ISO885910</td>
+</tr>
+<tr class="even">
+<td>LATIN7</td>
+<td>ISO 8859-13</td>
+<td>Baltic</td>
+<td>Yes</td>
+<td>1</td>
+<td>ISO885913</td>
+</tr>
+<tr class="odd">
+<td>LATIN8</td>
+<td>ISO 8859-14</td>
+<td>Celtic</td>
+<td>Yes</td>
+<td>1</td>
+<td>ISO885914</td>
+</tr>
+<tr class="even">
+<td>LATIN9</td>
+<td>ISO 8859-15</td>
+<td>LATIN1 with Euro and accents</td>
+<td>Yes</td>
+<td>1</td>
+<td>ISO885915</td>
+</tr>
+<tr class="odd">
+<td>LATIN10</td>
+<td>ISO 8859-16, ASRO SR 14111</td>
+<td>Romanian</td>
+<td>Yes</td>
+<td>1</td>
+<td>ISO885916</td>
+</tr>
+<tr class="even">
+<td>MULE_INTERNAL</td>
+<td>Mule internal code</td>
+<td>Multilingual Emacs</td>
+<td>Yes</td>
+<td>1-4</td>
+<td> </td>
+</tr>
+<tr class="odd">
+<td>SJIS</td>
+<td>Shift JIS</td>
+<td>Japanese</td>
+<td>No</td>
+<td>1-2</td>
+<td>Mskanji, ShiftJIS, WIN932, Windows932</td>
+</tr>
+<tr class="even">
+<td>SQL_ASCII</td>
+<td>unspecified2</td>
+<td>any</td>
+<td>No</td>
+<td>1</td>
+<td> </td>
+</tr>
+<tr class="odd">
+<td>UHC</td>
+<td>Unified Hangul Code</td>
+<td>Korean</td>
+<td>No</td>
+<td>1-2</td>
+<td>WIN949, Windows949</td>
+</tr>
+<tr class="even">
+<td>UTF8</td>
+<td>Unicode, 8-bit </td>
+<td>all</td>
+<td>Yes</td>
+<td>1-4</td>
+<td>Unicode</td>
+</tr>
+<tr class="odd">
+<td>WIN866</td>
+<td>Windows CP866</td>
+<td>Cyrillic</td>
+<td>Yes</td>
+<td>1</td>
+<td>ALT</td>
+</tr>
+<tr class="even">
+<td>WIN874</td>
+<td>Windows CP874</td>
+<td>Thai</td>
+<td>Yes</td>
+<td>1</td>
+<td> </td>
+</tr>
+<tr class="odd">
+<td>WIN1250</td>
+<td>Windows CP1250</td>
+<td>Central European</td>
+<td>Yes</td>
+<td>1</td>
+<td> </td>
+</tr>
+<tr class="even">
+<td>WIN1251</td>
+<td>Windows CP1251</td>
+<td>Cyrillic</td>
+<td>Yes</td>
+<td>1</td>
+<td>WIN</td>
+</tr>
+<tr class="odd">
+<td>WIN1252</td>
+<td>Windows CP1252</td>
+<td>Western European</td>
+<td>Yes</td>
+<td><p>1</p></td>
+<td> </td>
+</tr>
+<tr class="even">
+<td>WIN1253</td>
+<td>Windows CP1253</td>
+<td>Greek</td>
+<td>Yes</td>
+<td>1</td>
+<td> </td>
+</tr>
+<tr class="odd">
+<td>WIN1254</td>
+<td>Windows CP1254</td>
+<td>Turkish</td>
+<td>Yes</td>
+<td>1</td>
+<td> </td>
+</tr>
+<tr class="even">
+<td>WIN1255</td>
+<td>Windows CP1255</td>
+<td>Hebrew</td>
+<td>Yes</td>
+<td>1</td>
+<td> </td>
+</tr>
+<tr class="odd">
+<td>WIN1256</td>
+<td>Windows CP1256</td>
+<td>Arabic</td>
+<td>Yes</td>
+<td>1</td>
+<td> </td>
+</tr>
+<tr class="even">
+<td>WIN1257</td>
+<td>Windows CP1257</td>
+<td>Baltic</td>
+<td>Yes</td>
+<td>1</td>
+<td> </td>
+</tr>
+<tr class="odd">
+<td>WIN1258</td>
+<td>Windows CP1258</td>
+<td>Vietnamese</td>
+<td>Yes</td>
+<td>1</td>
+<td>ABC, TCVN, TCVN5712, VSCII </td>
+</tr>
+</tbody>
+</table>
+
+**Note:**
+
+- Not all the APIs support all the listed character sets. For example, the JDBC driver does not support MULE\_INTERNAL, LATIN6, LATIN8, and LATIN10.
+- The SQLASCII setting behaves considerable differently from the other settings. Byte values 0-127 are interpreted according to the ASCII standard, while byte values 128-255 are taken as uninterpreted characters. If you are working with any nonASCII data, it is unwise to use the SQL\_ASCII setting as a client encoding. SQL\_ASCII is not supported as a server encoding.
+
+## Setting the Character Set<a id="settingthecharacterset"></a>
+
+`hawq init` defines the default character set for a HAWQ system by reading the setting of the ENCODING parameter in the gp\_init\_config file at initialization time. The default character set is UNICODE or UTF8.
+
+You can create a database with a different character set besides what is used as the system-wide default. For example:
+
+``` pre
+CREATE DATABASE korean WITH ENCODING 'EUC_KR';
+```
+
+**Note:** Although you can specify any encoding you want for a database, it is unwise to choose an encoding that is not what is expected by the locale you have selected. The LC\_COLLATE and LC\_CTYPE settings imply a particular encoding, and locale-dependent operations (such as sorting) are likely to misinterpret data that is in an incompatible encoding.
+
+Since these locale settings are frozen by hawq init, the apparent flexibility to use different encodings in different databases is more theoretical than real.
+
+One way to use multiple encodings safely is to set the locale to C or POSIX during initialization time, thus disabling any real locale awareness.
+
+## Character Set Conversion Between Server and Client<a id="charactersetconversionbetweenserverandclient"></a>
+
+HAWQ supports automatic character set conversion between server and client for certain character set combinations. The conversion information is stored in the master pg\_conversion system catalog table. HAWQ comes with some predefined conversions or you can create a new conversion using the SQL command CREATE CONVERSION.
+
+| Server Character Set | Available Client Sets |
+|----------------------|--------------------------------------------------------------------------------------------------------------------------------|
+| BIG5 | not supported as a server encoding |
+| EUC\_CN | EUC\_CN, MULE\_INTERNAL, UTF8 |
+| EUC\_JP | EUC\_JP, MULE\_INTERNAL, SJIS, UTF8 |
+| EUC\_KR | EUC\_KR, MULE\_INTERNAL, UTF8 |
+| EUC\_TW | EUC\_TW, BIG5, MULE\_INTERNAL, UTF8 |
+| GB18030 | not supported as a server encoding |
+| GBK | not supported as a server encoding |
+| ISO\_8859\_5 | ISO\_8859\_5, KOI8, MULE\_INTERNAL, UTF8, WIN866, WIN1251 |
+| ISO\_8859\_6 | ISO\_8859\_6, UTF8 |
+| ISO\_8859\_7 | ISO\_8859\_7, UTF8 |
+| ISO\_8859\_8 | ISO\_8859\_8, UTF8 |
+| JOHAB | JOHAB, UTF8 |
+| KOI8 | KOI8, ISO\_8859\_5, MULE\_INTERNAL, UTF8, WIN866, WIN1251 |
+| LATIN1 | LATIN1, MULE\_INTERNAL, UTF8 |
+| LATIN2 | LATIN2, MULE\_INTERNAL, UTF8, WIN1250 |
+| LATIN3 | LATIN3, MULE\_INTERNAL, UTF8 |
+| LATIN4 | LATIN4, MULE\_INTERNAL, UTF8 |
+| LATIN5 | LATIN5, UTF8 |
+| LATIN6 | LATIN6, UTF8 |
+| LATIN7 | LATIN7, UTF8 |
+| LATIN8 | LATIN8, UTF8 |
+| LATIN9 | LATIN9, UTF8 |
+| LATIN10 | LATIN10, UTF8 |
+| MULE\_INTERNAL | MULE\_INTERNAL, BIG5, EUC\_CN, EUC\_JP, EUC\_KR, EUC\_TW, ISO\_8859\_5, KOI8, LATIN1 to LATIN4, SJIS, WIN866, WIN1250, WIN1251 |
+| SJIS | not supported as a server encoding |
+| SQL\_ASCII | not supported as a server encoding |
+| UHC | not supported as a server encoding |
+| UTF8 | all supported encodings |
+| WIN866 | WIN866 |
+| WIN874 | WIN874, UTF8 |
+| WIN1250 | WIN1250, LATIN2, MULE\_INTERNAL, UTF8 |
+| WIN1251 | WIN1251, ISO\_8859\_5, KOI8, MULE\_INTERNAL, UTF8, WIN866 |
+| WIN1252 | WIN1252, UTF8 |
+| WIN1253 | WIN1253, UTF8 |
+| WIN1254 | WIN1254, UTF8 |
+| WIN1255 | WIN1255, UTF8 |
+| WIN1256 | WIN1256, UTF8 |
+| WIN1257 | WIN1257, UTF8 |
+| WIN1258 | WIN1258, UTF8 |
+
+To enable automatic character set conversion, you have to tell HAWQ the character set (encoding) you would like to use in the client. There are several ways to accomplish this:
+
+- Using the \\encoding command in psql, which allows you to change client encoding on the fly.
+- Using SET client\_encoding TO. Setting the client encoding can be done with this SQL command:
+
+ ``` pre
+ SET CLIENT_ENCODING TO 'value';
+ ```
+
+ To query the current client encoding:
+
+ ``` pre
+ SHOW client_encoding;
+ ```
+
+ To return the default encoding:
+
+ ``` pre
+ RESET client_encoding;
+ ```
+
+- Using the PGCLIENTENCODING environment variable. When PGCLIENTENCODING is defined in the client's environment, that client encoding is automatically selected when a connection to the server is made. (This can subsequently be overridden using any of the other methods mentioned above.)
+- Setting the configuration parameter client\_encoding. If client\_encoding is set in the master `hawq-site.xml` file, that client encoding is automatically selected when a connection to HAWQ is made. (This can subsequently be overridden using any of the other methods mentioned above.)
+
+If the conversion of a particular character is not possible — suppose you chose EUC\_JP for the server and LATIN1 for the client, then some Japanese characters do not have a representation in LATIN1 — then an error is reported.
+
+If the client character set is defined as SQL\_ASCII, encoding conversion is disabled, regardless of the server’s character set. The use of SQL\_ASCII is unwise unless you are working with all-ASCII data. SQL\_ASCII is not supported as a server encoding.
+
+
diff --git a/reference/HAWQDataTypes.html.md.erb b/reference/HAWQDataTypes.html.md.erb
new file mode 100644
index 0000000..d0fd7bc
--- /dev/null
+++ b/reference/HAWQDataTypes.html.md.erb
@@ -0,0 +1,133 @@
+---
+title: Data Types
+---
+
+This topic provides a reference of the data types supported in HAWQ.
+
+HAWQ has a rich set of native data types available to users. Users may also define new data types using the `CREATE TYPE` command. This reference shows all of the built-in data types. In addition to the types listed here, there are also some internally used data types, such as **oid** (object identifier), but those are not documented in this guide.
+
+The following data types are specified by SQL:
+
+- bit
+- bit varying
+- boolean
+- character varying
+- char
+- character
+- date
+- decimal
+- double precision
+- integer
+- interval
+- numeric
+- real
+- smallint
+- time (with or without time zone)
+- timestamp (with or without time zone)
+- varchar
+
+Each data type has an external representation determined by its input and output functions. Many of the built-in types have obvious external formats. However, several types are unique to HAWQ, such as geometric paths, or have several possibilities for formats, such as the date and time types. Some of the input and output functions are not invertible. That is, the result of an output function may lose accuracy when compared to the original input.
+
+
+ <span class="tablecap">Table 1. HAWQ Built-in Data Types</span>
+=======
+
+
+| Name | Alias | Size | Range | Description |
+|--------------------------------------------|---------------------|-----------------------|---------------------------------------------|-----------------------------------------------------------------------------------|
+| bigint | int8 | 8 bytes | -9223372036854775808 to 9223372036854775807 | large range integer |
+| bigserial | serial8 | 8 bytes | 1 to 9223372036854775807 | large autoincrementing integer |
+| bit \[ (n) \] | | n bits | bit string constant | fixed-length bit string |
+| bit varying \[ (n) \] | varbit | actual number of bits | bit string constant | variable-length bit string |
+| boolean | bool | 1 byte | true/false, t/f, yes/no, y/n, 1/0 | logical Boolean (true/false) |
+| box | | 32 bytes | ((x1,y1),(x2,y2)) | rectangular box in the plane - not allowed in distribution key columns. |
+| bytea | | 1 byte + binarystring | sequence of octets | variable-length binary string |
+| character \[ (n) \] | char \[ (n) \] | 1 byte + n | strings up to n characters in length | fixed-length, blank padded |
+| character varying \[ (n) \] | varchar \[ (n) \] | 1 byte + binarystring | strings up to n characters in length | variable-length with limit |
+| cidr | | 12 or 24 bytes | | IPv4 networks |
+| circle | | 24 bytes | <(x,y),r> (center and radius) | circle in the plane - not allowed in distribution key columns. |
+| date | | 4 bytes | 4713 BC - 294,277 AD | calendar date (year, month, day) |
+| decimal \[ (p, s) \] | numeric \[ (p,s) \] | variable | no limit | user-specified, inexact |
+| double precision | float8 float | 8 bytes | 15 decimal digits precision | variable-precision, inexact |
+| inet | | 12 or 24 bytes | | IPv4 hosts and networks |
+| integer | int, int4 | 4 bytes | -2147483648 to +2147483647 | usual choice for integer |
+| interval \[ (p) \] | | 12 bytes | -178000000 years - 178000000 years | time span |
+| lseg | | 32 bytes | ((x1,y1),(x2,y2)) | line segment in the plane - not allowed in distribution key columns. |
+| macaddr | | 6 bytes | | MAC addresses |
+| money | | 4 bytes | -21474836.48 to +21474836.47 | currency amount |
+| path | | 16+16n bytes | \[(x1,y1),...\] | geometric path in the plane - not allowed in distribution key columns. |
+| point | | 16 bytes | (x, y) | geometric path in the plane - not allowed in distribution key columns. |
+| polygon | | 40+16n bytes | \[(x1,y1),...\] | closed geometric path in the plane - not allowed in the distribution key columns. |
+| real | float4 | 4 bytes | 6 decimal digits precision | variable-precision, inexact |
+| serial | serial4 | 4 bytes | 1 to 2147483647 | autoincrementing integer |
+| smallint | int2 | 2 bytes | -32768 to +32767 | small range integer |
+| text | | 1 byte + string size | strings of any length | variable unlimited length |
+| time \[ (p) \] \[ without time zone \] | | 8 bytes | 00:00:00\[.000000\] - 24:00:00\[.000000\] | time of day only |
+| time \[ (p) \] with time zone | timetz | 12 bytes | 00:00:00+1359 - 24:00:00-1359 | time of day only, with time zone |
+| timestamp \[ (p) \] \[without time zone \] | | 8 bytes | 4713 BC - 294,277 AD | both date and time |
+| timestamp \[ (p) \] with time zone | timestamptz | 8 bytes | 4713 BC - 294,277 AD | both date and time, with time zone |
+| xml | | 1 byte + xml size | xml of any length | variable unlimited length |
+
+
+For variable length data types (such as char, varchar, text, xml, etc.) if the data is greater than or equal to 127 bytes, the storage overhead is 4 bytes instead of 1.
+
+## Time Zones<a id="timezones"></a>
+
+Time zones, and time-zone conventions, are influenced by political decisions, not just earth geometry. Time zones around the world became somewhat standardized during the 1900's, but continue to be prone to arbitrary changes, particularly with respect to daylight-savings rules. HAWQ uses the widely-used zoneinfo time zone database for information about historical time zone rules. For times in the future, the assumption is that the latest known rules for a given time zone will continue to be observed indefinitely far into the future.
+
+HAWQ is compatible with the SQL standard definitions for typical usage. However, the SQL standard has an odd mix of date and time types and capabilities. Two obvious problems are:
+
+- Although the date type cannot have an associated time zone, the time type can. Time zones in the real world have little meaning unless associated with a date as well as a time, since the offset can vary through the year with daylight-saving time boundaries.
+- The default time zone is specified as a constant numeric offset from UTC. It is therefore impossible to adapt to daylight-saving time when doing date/time arithmetic across DST boundaries.
+
+To address these difficulties, use date/time types that contain both date and time when using time zones. Do not use the type time with time zone (although HAWQ supports this for legacy applications and for compliance with the SQL standard). HAWQ assumes your local time zone for any type containing only date or time.
+
+All timezone-aware dates and times are stored internally in UTC. They are converted to local time in the zone specified by the timezone configuration parameter before being displayed to the client.
+
+HAWQ allows you to specify time zones in three different forms:
+
+- A full time zone name, for example America/New\_York. HAWQ uses the widely-used zoneinfo time zone data for this purpose, so the same names are also recognized by much other software.
+- A time zone abbreviation, for example PST. Such a specification merely defines a particular offset from UTC, in contrast to full time zone names which can imply a set of daylight savings transition-date rules as well. You cannot set the configuration parameters timezone or log\_timezone to a time zone abbreviation, but you can use abbreviations in date/time input values and with the AT TIME ZONE operator.
+- In addition to the timezone names and abbreviations, HAWQ /> accepts POSIX-style time zone specifications of the form STDoffset or STDoffsetDST, where STD is a zone abbreviation, offset is a numeric offset in hours west from UTC, and DST is an optional daylight-savings zone abbreviation, assumed to stand for one hour ahead of the given offset. For example, if EST5EDT were not already a recognized zone name, it would be accepted and would be functionally equivalent to United States East Coast time. When a daylight-savings zone name is present, it is assumed to be used according to the same daylight-savings transition rules used in the zoneinfo time zone database's posixrules entry. In a standard HAWQ installation, posixrules is the same as US/Eastern, so that POSIX-style time zone specifications follow USA daylight-savings rules. If needed, you can adjust this behavior by replacing the posixrules file.
+
+In short, this is the difference between abbreviations and full names: abbreviations always represent a fixed offset from UTC, whereas most of the full names imply a local daylight-savings time rule, and so have two possible UTC offsets.
+
+One should be wary that the POSIX-style time zone feature can lead to silently accepting bogus input, since there is no check on the reasonableness of the zone abbreviations. For example, SET TIMEZONE TO FOOBAR0 will work, leaving the system effectively using a rather peculiar abbreviation for UTC. Another issue to keep in mind is that in POSIX time zone names, positive offsets are used for locations west of Greenwich. Everywhere else, PostgreSQL follows the ISO-8601 convention that positive timezone offsets are east of Greenwich.
+
+In all cases, timezone names are recognized case-insensitively.
+
+Neither full names nor abbreviations are hard-wired into the server, see [Date and Time Configuration Files](#dateandtimeconfigurationfiles).
+
+The timezone configuration parameter can be set in the file `hawq-site.xml`. There are also several special ways to set it:
+
+- If timezone is not specified in `hawq-site.xml` or as a server command-line option, the server attempts to use the value of the TZ environment variable as the default time zone. If TZ is not defined or is not any of the time zone names known to PostgreSQL, the server attempts to determine the operating system's default time zone by checking the behavior of the C library function localtime(). The default time zone is selected as the closest match from the known time zones.
+- The SQL command SET TIME ZONE sets the time zone for the session. This is an alternative spelling of SET TIMEZONE TO with a more SQL-spec-compatible syntax.
+- The PGTZ environment variable is used by libpq clients to send a SET TIME ZONE command to the server upon connection.
+
+## Date and Time Configuration Files<a id="dateandtimeconfigurationfiles"></a>
+
+Since timezone abbreviations are not well standardized, HAWQ /> provides a means to customize the set of abbreviations accepted by the server. The timezone\_abbreviations run-time parameter determines the active set of abbreviations. While this parameter can be altered by any database user, the possible values for it are under the control of the database administrator — they are in fact names of configuration files stored in .../share/timezonesets/ of the installation directory. By adding or altering files in that directory, the administrator can set local policy for timezone abbreviations.
+
+timezone\_abbreviations can be set to any file name found in .../share/timezonesets/, if the file's name is entirely alphabetic. (The prohibition against non-alphabetic characters in timezone\_abbreviations prevents reading files outside the intended directory, as well as reading editor backup files and other extraneous files.)
+
+A timezone abbreviation file can contain blank lines and comments beginning with \#. Non-comment lines must have one of these formats:
+
+``` pre
+time_zone_nameoffsettime_zone_nameoffset D
+@INCLUDE file_name
+@OVERRIDE
+```
+
+A time\_zone\_name is just the abbreviation being defined. The offset is the zone's offset in seconds from UTC, positive being east from Greenwich and negative being west. For example, -18000 would be five hours west of Greenwich, or North American east coast standard time. D indicates that the zone name represents local daylight-savings time rather than standard time. Since all known time zone offsets are on 15 minute boundaries, the number of seconds has to be a multiple of 900.
+
+The @INCLUDE syntax allows inclusion of another file in the .../share/timezonesets/ directory. Inclusion can be nested, to a limited depth.
+
+The @OVERRIDE syntax indicates that subsequent entries in the file can override previous entries (i.e., entries obtained from included files). Without this, conflicting definitions of the same timezone abbreviation are considered an error.
+
+In an unmodified installation, the file Default contains all the non-conflicting time zone abbreviations for most of the world. Additional files Australia and India are provided for those regions: these files first include the Default file and then add or modify timezones as needed.
+
+For reference purposes, a standard installation also contains files Africa.txt, America.txt, etc, containing information about every time zone abbreviation known to be in use according to the zoneinfo timezone database. The zone name definitions found in these files can be copied and pasted into a custom configuration file as needed.
+
+**Note:** These files cannot be directly referenced as timezone\_abbreviations settings, because of the dot embedded in their names.
+
+
diff --git a/reference/HAWQEnvironmentVariables.html.md.erb b/reference/HAWQEnvironmentVariables.html.md.erb
new file mode 100644
index 0000000..d4b754a
--- /dev/null
+++ b/reference/HAWQEnvironmentVariables.html.md.erb
@@ -0,0 +1,105 @@
+---
+title: Environment Variables
+---
+
+This topic contains a reference of the environment variables that you set for HAWQ.
+
+Set these in your user’s startup shell profile (such as `~/.bashrc` or `~/.bash_profile`), or in `/etc/profile`, if you want to set them for all users.
+
+## Required Environment Variables<a id="requiredenvironmentvariables"></a>
+
+**Note:** `GPHOME`, `PATH` and `LD_LIBRARY_PATH` can be set by sourcing the `greenplum_path.sh` file from your HAWQ installation directory.
+
+### GPHOME<a id="gphome"></a>
+
+This is the installed location of your HAWQ software. For example:
+
+``` pre
+GPHOME=/usr/local/hawq
+export GPHOME
+```
+
+### PATH<a id="path"></a>
+
+Your `PATH` environment variable should point to the location of the HAWQ bin directory. For example:
+
+``` pre
+PATH=$GPHOME/bin:$PATH
+export PATH
+```
+
+### LD\_LIBRARY\_PATH<a id="ld_library_path"></a>
+
+The `LD_LIBRARY_PATH` environment variable should point to the location of the `HAWQ/PostgreSQL` library files. For example:
+
+``` pre
+LD_LIBRARY_PATH=$GPHOME/lib
+export LD_LIBRARY_PATH
+```
+
+### MASTER\_DATA\_DIRECTORY<a id="master_data_directory"></a>
+
+This should point to the directory created by the `hawq init` utility in the master data directory location. For example:
+
+``` pre
+MASTER_DATA_DIRECTORY=/data/master
+export MASTER_DATA_DIRECTORY
+```
+
+## Optional Environment Variables<a id="optionalenvironmentvariables"></a>
+
+The following are HAWQ environment variables. You may want to add the connection-related environment variables to your profile, for convenience. That way, you do not have to type so many options on the command line for client connections. Note that these environment variables should be set on the HAWQ master host only.
+
+### PGAPPNAME<a id="pgappname"></a>
+
+This is the name of the application that is usually set by an application when it connects to the server. This name is displayed in the activity view and in log entries. The `PGAPPNAME` environmental variable behaves the same as the `application_name` connection parameter. The default value for `application_name` is `psql`. The name cannot be longer than 63 characters.
+
+### PGDATABASE<a id="pgdatabase"></a>
+
+The name of the default database to use when connecting.
+
+### PGHOST<a id="pghost"></a>
+
+The HAWQ master host name.
+
+### PGHOSTADDR<a id="pghostaddr"></a>
+
+The numeric IP address of the master host. This can be set instead of, or in addition to, `PGHOST`, to avoid DNS lookup overhead.
+
+### PGPASSWORD<a id="pgpassword"></a>
+
+The password used if the server demands password authentication. Use of this environment variable is not recommended, for security reasons (some operating systems allow non-root users to see process environment variables via ps). Instead, consider using the `~/.pgpass` file.
+
+### PGPASSFILE<a id="pgpassfile"></a>
+
+The name of the password file to use for lookups. If not set, it defaults to `~/.pgpass`.
+
+See The Password File under [Configuring Client Authentication](/20/clientaccess/client_auth.html).
+
+### PGOPTIONS<a id="pgoptions"></a>
+
+Sets additional configuration parameters for the HAWQ master server.
+
+### PGPORT<a id="pgport"></a>
+
+The port number of the HAWQ server on the master host. The default port is 5432.
+
+### PGUSER<a id="pguser"></a>
+
+The HAWQ user name used to connect.
+
+### PGDATESTYLE<a id="pgdatestyle"></a>
+
+Sets the default style of date/time representation for a session. (Equivalent to `SET datestyle TO....`)
+
+### PGTZ<a id="pgtz"></a>
+
+Sets the default time zone for a session. (Equivalent to `SET timezone TO....`)
+
+### PGCLIENTENCODING<a id="pgclientencoding"></a>
+
+Sets the default client character set encoding for a session. (Equivalent to `SET client_encoding TO....`)
+
+
+
+
diff --git a/reference/HAWQSampleSiteConfig.html.md.erb b/reference/HAWQSampleSiteConfig.html.md.erb
new file mode 100644
index 0000000..d4cae5a
--- /dev/null
+++ b/reference/HAWQSampleSiteConfig.html.md.erb
@@ -0,0 +1,120 @@
+---
+title: Sample hawq-site.xml Configuration File
+---
+
+```xml
+<configuration>
+ <property>
+ <name>default_hash_table_bucket_number</name>
+ <value>18</value>
+ </property>
+
+ <property>
+ <name>hawq_dfs_url</name>
+ <value>hawq.example.com:8020/hawq_default</value>
+ </property>
+
+ <property>
+ <name>hawq_global_rm_type</name>
+ <value>none</value>
+ </property>
+
+ <property>
+ <name>hawq_master_address_host</name>
+ <value>hawq.example.com</value>
+ </property>
+
+ <property>
+ <name>hawq_master_address_port</name>
+ <value>5432</value>
+ </property>
+
+ <property>
+ <name>hawq_master_directory</name>
+ <value>/data/hawq/master</value>
+ </property>
+
+ <property>
+ <name>hawq_master_temp_directory</name>
+ <value>/tmp/hawq/master</value>
+ </property>
+
+ <property>
+ <name>hawq_re_cgroup_hierarchy_name</name>
+ <value>hawq</value>
+ </property>
+
+ <property>
+ <name>hawq_re_cgroup_mount_point</name>
+ <value>/sys/fs/cgroup</value>
+ </property>
+
+ <property>
+ <name>hawq_re_cpu_enable</name>
+ <value>false</value>
+ </property>
+
+ <property>
+ <name>hawq_rm_memory_limit_perseg</name>
+ <value>64GB</value>
+ </property>
+
+ <property>
+ <name>hawq_rm_nvcore_limit_perseg</name>
+ <value>16</value>
+ </property>
+
+ <property>
+ <name>hawq_rm_nvseg_perquery_limit</name>
+ <value>512</value>
+ </property>
+
+ <property>
+ <name>hawq_rm_nvseg_perquery_perseg_limit</name>
+ <value>6</value>
+ </property>
+
+ <property>
+ <name>hawq_rm_yarn_address</name>
+ <value>rm.example.com:8050</value>
+ </property>
+
+ <property>
+ <name>hawq_rm_yarn_app_name</name>
+ <value>hawq</value>
+ </property>
+
+ <property>
+ <name>hawq_rm_yarn_queue_name</name>
+ <value>default</value>
+ </property>
+
+ <property>
+ <name>hawq_rm_yarn_scheduler_address</name>
+ <value>rm.example.com:8030</value>
+ </property>
+
+ <property>
+ <name>hawq_segment_address_port</name>
+ <value>40000</value>
+ </property>
+
+ <property>
+ <name>hawq_segment_directory</name>
+ <value>/data/hawq/segment</value>
+ </property>
+
+ <property>
+ <name>hawq_segment_temp_directory</name>
+ <value>/tmp/hawq/segment</value>
+ </property>
+
+ <property>
+ <name>hawq_standby_address_host</name>
+ <value>standbyhost.example.com</value>
+ </property>
+
+</configuration>
+```
+
+
diff --git a/reference/HAWQSiteConfig.html.md.erb b/reference/HAWQSiteConfig.html.md.erb
new file mode 100644
index 0000000..7910606
--- /dev/null
+++ b/reference/HAWQSiteConfig.html.md.erb
@@ -0,0 +1,19 @@
+---
+title: Server Configuration Parameter Reference
+---
+
+This section describes all server configuration guc/parameters that are available in HAWQ.
+
+Configuration guc/parameters are located in `$GPHOME/etc/hawq-site.xml`. This configuration file resides on all HAWQ instances and can be modified by using the `hawq config` utility. You can use the same configuration file cluster-wide across both master and segments.
+
+**Note:** While `postgresql.conf` still exists in HAWQ, any parameters defined in `hawq-site.xml` will overwrite configurations in `postgresql.conf`. For this reason, we recommend that you only use `hawq-site.xml` to configure your HAWQ cluster.
+
+- **[About Server Configuration Parameters](../reference/guc/guc_config.html)**
+
+- **[Configuration Parameter Categories](../reference/guc/guc_category-list.html)**
+
+- **[Configuration Parameters](../reference/guc/parameter_definitions.html)**
+
+- **[Sample hawq-site.xml Configuration File](../reference/HAWQSampleSiteConfig.html)**
+
+
diff --git a/reference/HDFSConfigurationParameterReference.html.md.erb b/reference/HDFSConfigurationParameterReference.html.md.erb
new file mode 100644
index 0000000..ef0ba0f
--- /dev/null
+++ b/reference/HDFSConfigurationParameterReference.html.md.erb
@@ -0,0 +1,257 @@
+---
+title: HDFS Configuration Reference
+---
+
+This reference page describes HDFS configuration values that are configured for HAWQ either within `hdfs-site.xml`, `core-site.xml`, or `hdfs-client.xml`.
+
+## HDFS Site Configuration (hdfs-site.xml and core-site.xml)<a id="topic_ixj_xw1_1w"></a>
+
+This topic provides a reference of the HDFS site configuration values recommended for HAWQ installations. These parameters are located in either `hdfs-site.xml` or `core-site.xml` of your HDFS deployment.
+
+This table describes the configuration parameters and values that are recommended for HAWQ installations. Only HDFS parameters that need to be modified or customized for HAWQ are listed.
+
+| Parameter | Description | Recommended Value for HAWQ Installs | Comments |
+|-------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `dfs.allow.truncate` | Allows truncate. | true | HAWQ requires that you enable `dfs.allow.truncate`. The HAWQ service will fail to start if `dfs.allow.truncate` is not set to `true`. |
+| `dfs.block.access.token.enable` | If `true`, access tokens are used as capabilities for accessing datanodes. If `false`, no access tokens are checked on accessing datanodes. | *false* for an unsecured HDFS cluster, or *true* for a secure cluster | |
+| `dfs.block.local-path-access.user` | Comma separated list of the users allowed to open block files on legacy short-circuit local read. | gpadmin | |
+| `dfs.client.read.shortcircuit` | This configuration parameter turns on short-circuit local reads. | true | In Ambari, this parameter corresponds to **HDFS Short-circuit read**. The value for this parameter should be the same in `hdfs-site.xml` and HAWQ's `hdfs-client.xml`. |
+| `dfs.client.socket-timeout` | The amount of time before a client connection times out when establishing a connection or reading. The value is expressed in milliseconds. | 300000000 | |
+| `dfs.client.use.legacy.blockreader.local` | Setting this value to false specifies that the new version of the short-circuit reader is used. Setting this value to true means that the legacy short-circuit reader would be used. | false | |
+| `dfs.datanode.data.dir.perm` | Permissions for the directories on on the local filesystem where the DFS data node store its blocks. The permissions can either be octal or symbolic. | 750 | In Ambari, this parameter corresponds to **DataNode directories permission** |
+| `dfs.datanode.handler.count` | The number of server threads for the datanode. | 60 | |
+| `dfs.datanode.max.transfer.threads` | Specifies the maximum number of threads to use for transferring data in and out of the DataNode. | 40960 | In Ambari, this parameter corresponds to **DataNode max data transfer threads** |
+| `dfs.datanode.socket.write.timeout` | The amount of time before a write operation times out, expressed in milliseconds. | 7200000 | |
+| `dfs.domain.socket.path` | (Optional.) The path to a UNIX domain socket to use for communication between the DataNode and local HDFS clients. If the string "\_PORT" is present in this path, it is replaced by the TCP port of the DataNode. | | If set, the value for this parameter should be the same in `hdfs-site.xml` and HAWQ's `hdfs-client.xml`. |
+| `dfs.namenode.accesstime.precision` | The access time for HDFS file is precise up to this value. Setting a value of 0 disables access times for HDFS. | 0 | In Ambari, this parameter corresponds to **Access time precision** |
+| `dfs.namenode.handler.count` | The number of server threads for the NameNode. | 600 | |
+| `dfs.support.append` | Whether HDFS is allowed to append to files. | true | |
+| `ipc.client.connection.maxidletime` | The maximum time in milliseconds after which a client will bring down the connection to the server. | 3600000 | In core-site.xml |
+| `ipc.client.connect.timeout` | Indicates the number of milliseconds a client will wait for the socket to establish a server connection. | 300000 | In core-site.xml |
+| `ipc.server.listen.queue.size` | Indicates the length of the listen queue for servers accepting client connections. | 3300 | In core-site.xml |
+
+## HDFS Client Configuration (hdfs-client.xml)<a id="topic_l1c_zw1_1w"></a>
+
+This topic provides a reference of the HAWQ configuration values located in `$GPHOME/etc/hdfs-client.xml`.
+
+This table describes the configuration parameters and their default values:
+
+<table>
+<colgroup>
+<col width="25%" />
+<col width="25%" />
+<col width="25%" />
+<col width="25%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Parameter</th>
+<th>Description</th>
+<th>Default Value</th>
+<th>Comments</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td><code class="ph codeph">dfs.client.failover.max.attempts</code></td>
+<td>The maximum number of times that the DFS client retries issuing a RPC call when multiple NameNodes are configured.</td>
+<td>15</td>
+<td> </td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">dfs.client.log.severity</code></td>
+<td>The minimal log severity level. Valid values include: FATAL, ERROR, INFO, DEBUG1, DEBUG2, and DEBUG3.</td>
+<td>INFO</td>
+<td> </td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">dfs.client.read.shortcircuit</code></td>
+<td>Determines whether the DataNode is bypassed when reading file blocks, if the block and client are on the same node. The default value, true, bypasses the DataNode.</td>
+<td>true</td>
+<td>The value for this parameter should be the same in <code class="ph codeph">hdfs-site.xml</code> and HAWQ's <code class="ph codeph">hdfs-client.xml</code>.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">dfs.client.use.legacy.blockreader.local</code></td>
+<td>Determines whether the legacy short-circuit reader implementation, based on HDFS-2246, is used. Set this property to true on non-Linux platforms that do not have the new implementation based on HDFS-347.</td>
+<td>false</td>
+<td> </td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">dfs.default.blocksize</code></td>
+<td>Default block size, in bytes.</td>
+<td>134217728</td>
+<td>Default is equivalent to 128 MB. </td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">dfs.default.replica</code></td>
+<td>The default number of replicas.</td>
+<td>3</td>
+<td> </td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">dfs.domain.socket.path</code></td>
+<td>(Optional.) The path to a UNIX domain socket to use for communication between the DataNode and local HDFS clients. If the string "_PORT" is present in this path, it is replaced by the TCP port of the DataNode.</td>
+<td> </td>
+<td>If set, the value for this parameter should be the same in <code class="ph codeph">hdfs-site.xml</code> and HAWQ's <code class="ph codeph">hdfs-client.xml</code>.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">dfs.prefetchsize</code></td>
+<td>The number of blocks for which information is pre-fetched.</td>
+<td>10</td>
+<td> </td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">hadoop.security.authentication</code></td>
+<td>Specifies the type of RPC authentication to use. A value of <code class="ph codeph">simple</code> indicates no authentication. A value of <code class="ph codeph">kerberos</code> enables authentication by Kerberos.</td>
+<td>simple</td>
+<td> </td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">input.connect.timeout</code></td>
+<td>The timeout interval, in milliseconds, for when the input stream is setting up a connection to a DataNode.</td>
+<td>600000</td>
+<td> Default is equal to 1 hour.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">input.localread.blockinfo.cachesize</code></td>
+<td>The size of the file block path information cache, in bytes.</td>
+<td>1000</td>
+<td> </td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">input.localread.default.buffersize</code></td>
+<td>The size of the buffer, in bytes, used to hold data from the file block and verify the checksum. This value is used only when <code class="ph codeph">dfs.client.read.shortcircuit</code> is set to true.</td>
+<td>1048576</td>
+<td>Default is equal to 1MB. Only used when is set to true.
+<p>If an older version of <code class="ph codeph">hdfs-client.xml</code> is retained during upgrade, to avoid performance degradation, set the <code class="ph codeph">input.localread.default.buffersize</code> to 2097152. </p></td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">input.read.getblockinfo.retry</code></td>
+<td>The maximum number of times the client should retry getting block information from the NameNode.</td>
+<td>3</td>
+<td></td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">input.read.timeout</code></td>
+<td>The timeout interval, in milliseconds, for when the input stream is reading from a DataNode.</td>
+<td>3600000</td>
+<td>Default is equal to 1 hour.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">input.write.timeout</code></td>
+<td>The timeout interval, in milliseconds, for when the input stream is writing to a DataNode.</td>
+<td>3600000</td>
+<td> </td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">output.close.timeout</code></td>
+<td>The timeout interval for closing an output stream, in milliseconds.</td>
+<td>900000</td>
+<td>Default is equal to 1.5 hours.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">output.connect.timeout</code></td>
+<td>The timeout interval, in milliseconds, for when the output stream is setting up a connection to a DataNode.</td>
+<td>600000</td>
+<td>Default is equal to 10 minutes.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">output.default.chunksize</code></td>
+<td>The chunk size of the pipeline, in bytes.</td>
+<td>512</td>
+<td> </td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">output.default.packetsize</code></td>
+<td>The packet size of the pipeline, in bytes.</td>
+<td>65536</td>
+<td>Default is equal to 64KB. </td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">output.default.write.retry</code></td>
+<td>The maximum number of times that the client should reattempt to set up a failed pipeline.</td>
+<td>10</td>
+<td> </td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">output.packetpool.size</code></td>
+<td>The maximum number of packets in a file's packet pool.</td>
+<td>1024</td>
+<td> </td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">output.read.timeout</code></td>
+<td>The timeout interval, in milliseconds, for when the output stream is reading from a DataNode.</td>
+<td>3600000</td>
+<td>Default is equal to 1 hour. </td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">output.replace-datanode-on-failure</code></td>
+<td>Determines whether the client adds a new DataNode to pipeline if the number of nodes in the pipeline is less than the specified number of replicas.</td>
+<td>false (if # of nodes less than or equal to 4), otherwise true</td>
+<td>When you deploy a HAWQ cluster, the <code class="ph codeph">hawq init</code> utility detects the number of nodes in the cluster and updates this configuration parameter accordingly. However, when expanding an existing cluster to 4 or more nodes, you must manually set this value to true. Set to false if you remove existing nodes and fall under 4 nodes.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">output.write.timeout</code></td>
+<td>The timeout interval, in milliseconds, for when the output stream is writing to a DataNode.</td>
+<td>3600000</td>
+<td>Default is equal to 1 hour.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">rpc.client.connect.retry</code></td>
+<td>The maximum number of times to retry a connection if the RPC client fails connect to the server.</td>
+<td>10</td>
+<td> </td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">rpc.client.connect.tcpnodelay</code></td>
+<td>Determines whether TCP_NODELAY is used when connecting to the RPC server.</td>
+<td>true</td>
+<td> </td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">rpc.client.connect.timeout</code></td>
+<td>The timeout interval for establishing the RPC client connection, in milliseconds.</td>
+<td>600000</td>
+<td>Default equals 10 minutes.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">rpc.client.max.idle</code></td>
+<td>The maximum idle time for an RPC connection, in milliseconds.</td>
+<td>10000</td>
+<td>Default equals 10 seconds.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">rpc.client.ping.interval</code></td>
+<td>The interval which the RPC client send a heart beat to server. 0 means disable.</td>
+<td>10000</td>
+<td> </td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">rpc.client.read.timeout</code></td>
+<td>The timeout interval, in milliseconds, for when the RPC client is reading from the server.</td>
+<td>3600000</td>
+<td>Default equals 1 hour.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">rpc.client.socket.linger.timeout</code></td>
+<td>The value to set for the SO_LINGER socket when connecting to the RPC server.</td>
+<td>-1</td>
+<td> </td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">rpc.client.timeout</code></td>
+<td>The timeout interval of an RPC invocation, in milliseconds.</td>
+<td>3600000</td>
+<td>Default equals 1 hour.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">rpc.client.write.timeout</code></td>
+<td>The timeout interval, in milliseconds, for when the RPC client is writing to the server.</td>
+<td>3600000</td>
+<td>Default equals 1 hour.</td>
+</tr>
+</tbody>
+</table>
+
+
diff --git a/reference/SQLCommandReference.html.md.erb b/reference/SQLCommandReference.html.md.erb
new file mode 100644
index 0000000..7bbe625
--- /dev/null
+++ b/reference/SQLCommandReference.html.md.erb
@@ -0,0 +1,163 @@
+---
+title: SQL Commands
+---
+
+This section contains a description and the syntax of the SQL commands supported by HAWQ.
+
+- **[ABORT](../reference/sql/ABORT.html)**
+
+- **[ALTER AGGREGATE](../reference/sql/ALTER-AGGREGATE.html)**
+
+- **[ALTER DATABASE](../reference/sql/ALTER-DATABASE.html)**
+
+- **[ALTER FUNCTION](../reference/sql/ALTER-FUNCTION.html)**
+
+- **[ALTER OPERATOR](../reference/sql/ALTER-OPERATOR.html)**
+
+- **[ALTER OPERATOR CLASS](../reference/sql/ALTER-OPERATOR-CLASS.html)**
+
+- **[ALTER RESOURCE QUEUE](../reference/sql/ALTER-RESOURCE-QUEUE.html)**
+
+- **[ALTER ROLE](../reference/sql/ALTER-ROLE.html)**
+
+- **[ALTER TABLE](../reference/sql/ALTER-TABLE.html)**
+
+- **[ALTER TABLESPACE](../reference/sql/ALTER-TABLESPACE.html)**
+
+- **[ALTER TYPE](../reference/sql/ALTER-TYPE.html)**
+
+- **[ALTER USER](../reference/sql/ALTER-USER.html)**
+
+- **[ANALYZE](../reference/sql/ANALYZE.html)**
+
+- **[BEGIN](../reference/sql/BEGIN.html)**
+
+- **[CHECKPOINT](../reference/sql/CHECKPOINT.html)**
+
+- **[CLOSE](../reference/sql/CLOSE.html)**
+
+- **[COMMIT](../reference/sql/COMMIT.html)**
+
+- **[COPY](../reference/sql/COPY.html)**
+
+- **[CREATE AGGREGATE](../reference/sql/CREATE-AGGREGATE.html)**
+
+- **[CREATE DATABASE](../reference/sql/CREATE-DATABASE.html)**
+
+- **[CREATE EXTERNAL TABLE](../reference/sql/CREATE-EXTERNAL-TABLE.html)**
+
+- **[CREATE FUNCTION](../reference/sql/CREATE-FUNCTION.html)**
+
+- **[CREATE GROUP](../reference/sql/CREATE-GROUP.html)**
+
+- **[CREATE LANGUAGE](../reference/sql/CREATE-LANGUAGE.html)**
+
+- **[CREATE OPERATOR](../reference/sql/CREATE-OPERATOR.html)**
+
+- **[CREATE OPERATOR CLASS](../reference/sql/CREATE-OPERATOR-CLASS.html)**
+
+- **[CREATE RESOURCE QUEUE](../reference/sql/CREATE-RESOURCE-QUEUE.html)**
+
+- **[CREATE ROLE](../reference/sql/CREATE-ROLE.html)**
+
+- **[CREATE SCHEMA](../reference/sql/CREATE-SCHEMA.html)**
+
+- **[CREATE SEQUENCE](../reference/sql/CREATE-SEQUENCE.html)**
+
+- **[CREATE TABLE](../reference/sql/CREATE-TABLE.html)**
+
+- **[CREATE TABLE AS](../reference/sql/CREATE-TABLE-AS.html)**
+
+- **[CREATE TABLESPACE](../reference/sql/CREATE-TABLESPACE.html)**
+
+- **[CREATE TYPE](../reference/sql/CREATE-TYPE.html)**
+
+- **[CREATE USER](../reference/sql/CREATE-USER.html)**
+
+- **[CREATE VIEW](../reference/sql/CREATE-VIEW.html)**
+
+- **[DEALLOCATE](../reference/sql/DEALLOCATE.html)**
+
+- **[DECLARE](../reference/sql/DECLARE.html)**
+
+- **[DROP AGGREGATE](../reference/sql/DROP-AGGREGATE.html)**
+
+- **[DROP DATABASE](../reference/sql/DROP-DATABASE.html)**
+
+- **[DROP EXTERNAL TABLE](../reference/sql/DROP-EXTERNAL-TABLE.html)**
+
+- **[DROP FILESPACE](../reference/sql/DROP-FILESPACE.html)**
+
+- **[DROP FUNCTION](../reference/sql/DROP-FUNCTION.html)**
+
+- **[DROP GROUP](../reference/sql/DROP-GROUP.html)**
+
+- **[DROP OPERATOR](../reference/sql/DROP-OPERATOR.html)**
+
+- **[DROP OPERATOR CLASS](../reference/sql/DROP-OPERATOR-CLASS.html)**
+
+- **[DROP OWNED](../reference/sql/DROP-OWNED.html)**
+
+- **[DROP RESOURCE QUEUE](../reference/sql/DROP-RESOURCE-QUEUE.html)**
+
+- **[DROP ROLE](../reference/sql/DROP-ROLE.html)**
+
+- **[DROP SCHEMA](../reference/sql/DROP-SCHEMA.html)**
+
+- **[DROP SEQUENCE](../reference/sql/DROP-SEQUENCE.html)**
+
+- **[DROP TABLE](../reference/sql/DROP-TABLE.html)**
+
+- **[DROP TABLESPACE](../reference/sql/DROP-TABLESPACE.html)**
+
+- **[DROP TYPE](../reference/sql/DROP-TYPE.html)**
+
+- **[DROP USER](../reference/sql/DROP-USER.html)**
+
+- **[DROP VIEW](../reference/sql/DROP-VIEW.html)**
+
+- **[END](../reference/sql/END.html)**
+
+- **[EXECUTE](../reference/sql/EXECUTE.html)**
+
+- **[EXPLAIN](../reference/sql/EXPLAIN.html)**
+
+- **[FETCH](../reference/sql/FETCH.html)**
+
+- **[GRANT](../reference/sql/GRANT.html)**
+
+- **[INSERT](../reference/sql/INSERT.html)**
+
+- **[PREPARE](../reference/sql/PREPARE.html)**
+
+- **[REASSIGN OWNED](../reference/sql/REASSIGN-OWNED.html)**
+
+- **[RELEASE SAVEPOINT](../reference/sql/RELEASE-SAVEPOINT.html)**
+
+- **[RESET](../reference/sql/RESET.html)**
+
+- **[REVOKE](../reference/sql/REVOKE.html)**
+
+- **[ROLLBACK](../reference/sql/ROLLBACK.html)**
+
+- **[ROLLBACK TO SAVEPOINT](../reference/sql/ROLLBACK-TO-SAVEPOINT.html)**
+
+- **[SAVEPOINT](../reference/sql/SAVEPOINT.html)**
+
+- **[SELECT](../reference/sql/SELECT.html)**
+
+- **[SELECT INTO](../reference/sql/SELECT-INTO.html)**
+
+- **[SET](../reference/sql/SET.html)**
+
+- **[SET ROLE](../reference/sql/SET-ROLE.html)**
+
+- **[SET SESSION AUTHORIZATION](../reference/sql/SET-SESSION-AUTHORIZATION.html)**
+
+- **[SHOW](../reference/sql/SHOW.html)**
+
+- **[TRUNCATE](../reference/sql/TRUNCATE.html)**
+
+- **[VACUUM](../reference/sql/VACUUM.html)**
+
+
diff --git a/reference/catalog/catalog_ref-html.html.md.erb b/reference/catalog/catalog_ref-html.html.md.erb
new file mode 100644
index 0000000..864f106
--- /dev/null
+++ b/reference/catalog/catalog_ref-html.html.md.erb
@@ -0,0 +1,143 @@
+---
+title: System Catalogs Definitions
+---
+
+System catalog table and view definitions in alphabetical order.
+
+- **[gp\_configuration\_history](../../reference/catalog/gp_configuration_history.html)**
+
+- **[gp\_distribution\_policy](../../reference/catalog/gp_distribution_policy.html)**
+
+- **[gp\_global\_sequence](../../reference/catalog/gp_global_sequence.html)**
+
+- **[gp\_master\_mirroring](../../reference/catalog/gp_master_mirroring.html)**
+
+- **[gp\_persistent\_database\_node](../../reference/catalog/gp_persistent_database_node.html)**
+
+- **[gp\_persistent\_filespace\_node](../../reference/catalog/gp_persistent_filespace_node.html)**
+
+- **[gp\_persistent\_relation\_node](../../reference/catalog/gp_persistent_relation_node.html)**
+
+- **[gp\_persistent\_relfile\_node](../../reference/catalog/gp_persistent_relfile_node.html)**
+
+- **[gp\_persistent\_tablespace\_node](../../reference/catalog/gp_persistent_tablespace_node.html)**
+
+- **[gp\_relfile\_node](../../reference/catalog/gp_relfile_node.html)**
+
+- **[gp\_segment\_configuration](../../reference/catalog/gp_segment_configuration.html)**
+
+- **[gp\_version\_at\_initdb](../../reference/catalog/gp_version_at_initdb.html)**
+
+- **[pg\_aggregate](../../reference/catalog/pg_aggregate.html)**
+
+- **[pg\_am](../../reference/catalog/pg_am.html)**
+
+- **[pg\_amop](../../reference/catalog/pg_amop.html)**
+
+- **[pg\_amproc](../../reference/catalog/pg_amproc.html)**
+
+- **[pg\_appendonly](../../reference/catalog/pg_appendonly.html)**
+
+- **[pg\_attrdef](../../reference/catalog/pg_attrdef.html)**
+
+- **[pg\_attribute](../../reference/catalog/pg_attribute.html)**
+
+- **[pg\_attribute\_encoding](../../reference/catalog/pg_attribute_encoding.html)**
+
+- **[pg\_auth\_members](../../reference/catalog/pg_auth_members.html)**
+
+- **[pg\_authid](../../reference/catalog/pg_authid.html)**
+
+- **[pg\_cast](../../reference/catalog/pg_cast.html)**
+
+- **[pg\_class](../../reference/catalog/pg_class.html)**
+
+- **[pg\_compression](../../reference/catalog/pg_compression.html)**
+
+- **[pg\_constraint](../../reference/catalog/pg_constraint.html)**
+
+- **[pg\_conversion](../../reference/catalog/pg_conversion.html)**
+
+- **[pg\_database](../../reference/catalog/pg_database.html)**
+
+- **[pg\_depend](../../reference/catalog/pg_depend.html)**
+
+- **[pg\_description](../../reference/catalog/pg_description.html)**
+
+- **[pg\_exttable](../../reference/catalog/pg_exttable.html)**
+
+- **[pg\_filespace](../../reference/catalog/pg_filespace.html)**
+
+- **[pg\_filespace\_entry](../../reference/catalog/pg_filespace_entry.html)**
+
+- **[pg\_index](../../reference/catalog/pg_index.html)**
+
+- **[pg\_inherits](../../reference/catalog/pg_inherits.html)**
+
+- **[pg\_language](../../reference/catalog/pg_language.html)**
+
+- **[pg\_largeobject](../../reference/catalog/pg_largeobject.html)**
+
+- **[pg\_listener](../../reference/catalog/pg_listener.html)**
+
+- **[pg\_locks](../../reference/catalog/pg_locks.html)**
+
+- **[pg\_namespace](../../reference/catalog/pg_namespace.html)**
+
+- **[pg\_opclass](../../reference/catalog/pg_opclass.html)**
+
+- **[pg\_operator](../../reference/catalog/pg_operator.html)**
+
+- **[pg\_partition](../../reference/catalog/pg_partition.html)**
+
+- **[pg\_partition\_columns](../../reference/catalog/pg_partition_columns.html)**
+
+- **[pg\_partition\_encoding](../../reference/catalog/pg_partition_encoding.html)**
+
+- **[pg\_partition\_rule](../../reference/catalog/pg_partition_rule.html)**
+
+- **[pg\_partition\_templates](../../reference/catalog/pg_partition_templates.html)**
+
+- **[pg\_partitions](../../reference/catalog/pg_partitions.html)**
+
+- **[pg\_pltemplate](../../reference/catalog/pg_pltemplate.html)**
+
+- **[pg\_proc](../../reference/catalog/pg_proc.html)**
+
+- **[pg\_resqueue](../../reference/catalog/pg_resqueue.html)**
+
+- **[pg\_resqueue\_status](../../reference/catalog/pg_resqueue_status.html)**
+
+- **[pg\_rewrite](../../reference/catalog/pg_rewrite.html)**
+
+- **[pg\_roles](../../reference/catalog/pg_roles.html)**
+
+- **[pg\_shdepend](../../reference/catalog/pg_shdepend.html)**
+
+- **[pg\_shdescription](../../reference/catalog/pg_shdescription.html)**
+
+- **[pg\_stat\_activity](../../reference/catalog/pg_stat_activity.html)**
+
+- **[pg\_stat\_last\_operation](../../reference/catalog/pg_stat_last_operation.html)**
+
+- **[pg\_stat\_last\_shoperation](../../reference/catalog/pg_stat_last_shoperation.html)**
+
+- **[pg\_stat\_operations](../../reference/catalog/pg_stat_operations.html)**
+
+- **[pg\_stat\_partition\_operations](../../reference/catalog/pg_stat_partition_operations.html)**
+
+- **[pg\_statistic](../../reference/catalog/pg_statistic.html)**
+
+- **[pg\_stats](../../reference/catalog/pg_stats.html)**
+
+- **[pg\_tablespace](../../reference/catalog/pg_tablespace.html)**
+
+- **[pg\_trigger](../../reference/catalog/pg_trigger.html)**
+
+- **[pg\_type](../../reference/catalog/pg_type.html)**
+
+- **[pg\_type\_encoding](../../reference/catalog/pg_type_encoding.html)**
+
+- **[pg\_window](../../reference/catalog/pg_window.html)**
+
+
diff --git a/reference/catalog/catalog_ref-tables.html.md.erb b/reference/catalog/catalog_ref-tables.html.md.erb
new file mode 100644
index 0000000..61aa936
--- /dev/null
+++ b/reference/catalog/catalog_ref-tables.html.md.erb
@@ -0,0 +1,68 @@
+---
+title: System Tables
+---
+
+This topic lists the system tables included in HAWQ.
+
+- gp\_configuration (Deprecated. See [gp\_segment\_configuration](gp_segment_configuration.html#topic1).)
+- [gp\_configuration\_history](gp_configuration_history.html#topic1)
+- [gp\_distribution\_policy](gp_distribution_policy.html#topic1)
+- [gp\_global\_sequence](gp_global_sequence.html#topic1)
+- [gp\_master\_mirroring](gp_master_mirroring.html#topic1)
+- [gp\_persistent\_database\_node](gp_persistent_database_node.html#topic1)
+- [gp\_persistent\_filespace\_node](gp_persistent_filespace_node.html#topic1)
+- [gp\_persistent\_relation\_node](gp_persistent_relation_node.html#topic1)
+- [gp\_persistent\_tablespace\_node](gp_persistent_tablespace_node.html#topic1)
+- gp\_relation\_node (Deprecated. See [gp\_relfile\_node](gp_relfile_node.html).)
+- [gp\_segment\_configuration](gp_segment_configuration.html#topic1)
+- [gp\_version\_at\_initdb](gp_version_at_initdb.html#topic1)
+- [pg\_aggregate](pg_aggregate.html#topic1)
+- [pg\_am](pg_am.html#topic1)
+- [pg\_amop](pg_amop.html#topic1)
+- [pg\_amproc](pg_amproc.html#topic1)
+- [pg\_appendonly](pg_appendonly.html#topic1)
+- [pg\_attrdef](pg_attrdef.html#topic1)
+- [pg\_attribute](pg_attribute.html#topic1)
+- [pg\_auth\_members](pg_auth_members.html#topic1)
+- [pg\_authid](pg_authid.html#topic1)
+- pg\_autovacuum (not supported)
+- [pg\_cast](pg_cast.html#topic1)
+- [pg\_class](pg_class.html#topic1)
+- [pg\_constraint](pg_constraint.html#topic1)
+- [pg\_conversion](pg_conversion.html#topic1)
+- [pg\_database](pg_database.html#topic1)
+- [pg\_depend](pg_depend.html#topic1)
+- [pg\_description](pg_description.html#topic1)
+- [pg\_exttable](pg_exttable.html#topic1)
+- [pg\_filespace](pg_filespace.html#topic1)
+- [pg\_filespace\_entry](pg_filespace_entry.html#topic1)
+- pg\_foreign\_data\_wrapper (not supported)
+- pg\_foreign\_server (not supported)
+- pg\_foreign\_table (not supported)
+- [pg\_index](pg_index.html#topic1)
+- [pg\_inherits](pg_inherits.html#topic1)
+- [pg\_language](pg_language.html#topic1)
+- [pg\_largeobject](pg_largeobject.html#topic1)
+- [pg\_listener](pg_listener.html#topic1)
+- [pg\_namespace](pg_namespace.html#topic1)
+- [pg\_opclass](pg_opclass.html#topic1)
+- [pg\_operator](pg_operator.html#topic1)
+- [pg\_partition](pg_partition.html#topic1)
+- [pg\_partition\_rule](pg_partition_rule.html#topic1)
+- [pg\_pltemplate](pg_pltemplate.html#topic1)
+- [pg\_proc](pg_proc.html#topic1)
+- [pg\_resqueue](pg_resqueue.html#topic1)
+- [pg\_rewrite](pg_rewrite.html#topic1)
+- [pg\_shdepend](pg_shdepend.html#topic1)
+- [pg\_shdescription](pg_shdescription.html#topic1)
+- [pg\_stat\_last\_operation](pg_stat_last_operation.html#topic1)
+- [pg\_stat\_last\_shoperation](pg_stat_last_shoperation.html#topic1)
+- [pg\_statistic](pg_statistic.html#topic1)
+- [pg\_stats](pg_stats.html#topic1)
+- [pg\_tablespace](pg_tablespace.html#topic1)
+- [pg\_trigger](pg_trigger.html#topic1)
+- [pg\_type](pg_type.html#topic1)
+- pg\_user\_mapping (not supported)
+- [pg\_window](pg_window.html#topic1)
+
+
diff --git a/reference/catalog/catalog_ref-views.html.md.erb b/reference/catalog/catalog_ref-views.html.md.erb
new file mode 100644
index 0000000..4a7ad74
--- /dev/null
+++ b/reference/catalog/catalog_ref-views.html.md.erb
@@ -0,0 +1,21 @@
+---
+title: System Views
+---
+
+HAWQ provides the following system views not available in PostgreSQL.
+
+- pg\_max\_external\_files (shows number of external table files allowed per segment host when using the file:// protocol)
+- [pg\_partition\_columns](pg_partition_columns.html#topic1)
+- [pg\_partition\_templates](pg_partition_templates.html#topic1)
+- [pg\_partitions](pg_partitions.html#topic1)
+- [pg\_stat\_activity](pg_stat_activity.html#topic1)
+- [pg\_resqueue\_status](pg_resqueue_status.html#topic1)
+- [pg\_stats](pg_stats.html#topic1)
+
+For more information about the standard system views supported in PostgreSQL and HAWQ, see the following sections of the PostgreSQL documentation:
+
+- [System Views](http://www.postgresql.org/docs/8.2/static/views-overview.html)
+- [Statistics Collector Views](http://www.postgresql.org/docs/8.2/static/monitoring-stats.html#MONITORING-STATS-VIEWS-TABLE)
+- [The Information Schema](http://www.postgresql.org/docs/8.2/static/information-schema.html)
+
+
diff --git a/reference/catalog/catalog_ref.html.md.erb b/reference/catalog/catalog_ref.html.md.erb
new file mode 100644
index 0000000..143091f
--- /dev/null
+++ b/reference/catalog/catalog_ref.html.md.erb
@@ -0,0 +1,21 @@
+---
+title: System Catalog Reference
+---
+
+This reference describes the HAWQ system catalog tables and views.
+
+System tables prefixed with '`gp_`' relate to the parallel features of HAWQ. Tables prefixed with '`pg_`' are either standard PostgreSQL system catalog tables supported in HAWQ, or are related to features HAWQ that provides to enhance PostgreSQL for data warehousing workloads. Note that the global system catalog for HAWQ resides on the master instance.
+
+- **[System Tables](../../reference/catalog/catalog_ref-tables.html)**
+
+ This topic lists the system tables included in HAWQ.
+
+- **[System Views](../../reference/catalog/catalog_ref-views.html)**
+
+ HAWQ provides the following system views not available in PostgreSQL.
+
+- **[System Catalogs Definitions](../../reference/catalog/catalog_ref-html.html)**
+
+ System catalog table and view definitions in alphabetical order.
+
+
diff --git a/reference/catalog/gp_configuration_history.html.md.erb b/reference/catalog/gp_configuration_history.html.md.erb
new file mode 100644
index 0000000..bfc9425
--- /dev/null
+++ b/reference/catalog/gp_configuration_history.html.md.erb
@@ -0,0 +1,23 @@
+---
+title: gp_configuration_history
+---
+
+The `gp_configuration_history` table contains information about system changes related to fault detection and recovery operations. The HAWQ [fault tolerance service](../../admin/FaultTolerance.html) logs data to this table, as do certain related management utilities such as `hawq init`. For example, when you add a new segment to the system, records for these events are logged to `gp_configuration_history`. If a segment is marked as down by the fault tolerance service in the [gp_segment_configuration](gp_segment_configuration.html) catalog table, then the reason for being marked as down is recorded in this table.
+
+The event descriptions stored in this table may be helpful for troubleshooting serious system issues in collaboration with HAWQ support technicians.
+
+This table is populated only on the master. This table is defined in the `pg_global` tablespace, meaning it is globally shared across all databases in the system.
+
+<a id="topic1__ev138428"></a>
+<span class="tablecap">Table 1. pg\_catalog.gp\_configuration\_history</span>
+
+| column | type | references | description |
+|---------------------|--------------------------|------------|----------------------------------------------------------------------------------|
+| `time` | timestamp with time zone | | Timestamp for the event recorded. |
+| `desc` | text | | Text description of the event. |
+| registration\_order | integer | | The registration order of a segment. May be changed after restarting the master. |
+| hostname | text | | The hostname of a segment |
+
+
+
+
diff --git a/reference/catalog/gp_distribution_policy.html.md.erb b/reference/catalog/gp_distribution_policy.html.md.erb
new file mode 100644
index 0000000..6b227e6
--- /dev/null
+++ b/reference/catalog/gp_distribution_policy.html.md.erb
@@ -0,0 +1,18 @@
+---
+title: gp_distribution_policy
+---
+
+The `gp_distribution_policy` table contains information about HAWQ tables and their policy for distributing table data across the segments. This table is populated only on the master. This table is not globally shared, meaning each database has its own copy of this table.
+
+<a id="topic1__ey142842"></a>
+<span class="tablecap">Table 1. pg\_catalog.gp\_distribution\_policy</span>
+
+| column | type | references | description |
+|-------------|--------------|----------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `localoid` | oid | pg\_class.oid | The table object identifier (OID). |
+| `attrnums` | smallint\[\] | pg\_attribute.attnum | The column number(s) of the distribution column(s). |
+| `bucketnum` | integer | | Number of hash buckets used in creating a hash-distributed table or for external table intermediate processing. The number of buckets also affects how many virtual segment are created when processing data. |
+
+
+
+
diff --git a/reference/catalog/gp_global_sequence.html.md.erb b/reference/catalog/gp_global_sequence.html.md.erb
new file mode 100644
index 0000000..ac2ff2f
--- /dev/null
+++ b/reference/catalog/gp_global_sequence.html.md.erb
@@ -0,0 +1,16 @@
+---
+title: gp_global_sequence
+---
+
+The `gp_global_sequence` table contains the log sequence number position in the transaction log. This table is used by persistent tables.
+
+<a id="topic1__fe138428"></a>
+<span class="tablecap">Table 1. pg\_catalog.gp\_global\_sequence</span>
+
+| column | type | references | description |
+|----------------|--------|------------|-----------------------------------------------------|
+| `sequence_num` | bigint | | Log sequence number position in the transaction log |
+
+>
+
+
diff --git a/reference/catalog/gp_master_mirroring.html.md.erb b/reference/catalog/gp_master_mirroring.html.md.erb
new file mode 100644
index 0000000..5cd6378
--- /dev/null
+++ b/reference/catalog/gp_master_mirroring.html.md.erb
@@ -0,0 +1,19 @@
+---
+title: gp_master_mirroring
+---
+
+The `gp_master_mirroring` table contains state information about the standby master host and its associated write-ahead log (WAL) replication process. If this synchronization process (`gpsyncagent`) fails on the standby master, it may not always be noticeable to users of the system. This catalog is a place where HAWQ administrators can check to see if the standby master is current and fully synchronized.
+
+<a id="topic1__op164584"></a>
+<span class="tablecap">Table 1. pg\_catalog.gp\_master\_mirroring</span>
+
+| column | type | references | description |
+|-----------------|--------------------------|------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `summary_state` | text | | The current state of the log replication process between the master and standby master - logs are either ‘Synchronized’ or ‘Not Synchronized’ |
+| `detail_state` | text | | If not synchronized, this column will have information about the cause of the error. |
+| `log_time` | timestamp with time zone | | This contains the timestamp of the last time a master mirroring change occurred. For example, the timestamp when the value of summary\_state changed from "Synchronized" to "Not Synchronized". If no changes occur with regards to the standby master (it stays synchronized), the timestamp is not updated. |
+| `error_message` | text | | If not synchronized, this column will have the error message from the failed synchronization attempt. |
+
+
+
+
diff --git a/reference/catalog/gp_persistent_database_node.html.md.erb b/reference/catalog/gp_persistent_database_node.html.md.erb
new file mode 100644
index 0000000..2a3537a
--- /dev/null
+++ b/reference/catalog/gp_persistent_database_node.html.md.erb
@@ -0,0 +1,71 @@
+---
+title: gp_persistent_database_node
+---
+
+The `gp_persistent_database_node` table keeps track of the status of file system objects in relation to the transaction status of database objects. This information is used to make sure the state of the system catalogs and the file system files persisted to disk are synchronized.
+
+<a id="topic1__fi138428"></a>
+<span class="tablecap">Table 1. pg\_catalog.gp\_persistent\_database\_node</span>
+
+<table>
+<colgroup>
+<col width="25%" />
+<col width="25%" />
+<col width="25%" />
+<col width="25%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>column</th>
+<th>type</th>
+<th>references</th>
+<th>description</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td><code class="ph codeph">tablespace_oid</code></td>
+<td>oid</td>
+<td>pg_tablespace.oid</td>
+<td>Table space object id.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">database_oid</code></td>
+<td>oid</td>
+<td>pg_database.oid</td>
+<td>Database object id.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">persistent_state</code></td>
+<td>smallint</td>
+<td> </td>
+<td>0 - free
+<p>1 - create pending</p>
+<p>2 - created</p>
+<p>3 - drop pending</p>
+<p>4 - aborting create</p>
+<p>5 - "Just in Time" create pending</p>
+<p>6 - bulk load create pending</p></td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">parent_xid</code></td>
+<td>integer</td>
+<td> </td>
+<td>Global transaction id.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">persistent_serial_num</code></td>
+<td>bigint</td>
+<td> </td>
+<td>Log sequence number position in the transaction log for a file block.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">previous_free_tid</code></td>
+<td>tid</td>
+<td> </td>
+<td>Used by HAWQ to internally manage persistent representations of file system objects.</td>
+</tr>
+</tbody>
+</table>
+
+
diff --git a/reference/catalog/gp_persistent_filespace_node.html.md.erb b/reference/catalog/gp_persistent_filespace_node.html.md.erb
new file mode 100644
index 0000000..b682a88
--- /dev/null
+++ b/reference/catalog/gp_persistent_filespace_node.html.md.erb
@@ -0,0 +1,83 @@
+---
+title: gp_persistent_filespace_node
+---
+
+The `gp_persistent_filespace_node` table keeps track of the status of file system objects in relation to the transaction status of filespace objects. This information is used to make sure the state of the system catalogs and the file system files persisted to disk are synchronized.
+
+<a id="topic1__fj138428"></a>
+<span class="tablecap">Table 1. pg\_catalog.gp\_persistent\_filespace\_node</span>
+
+<table>
+<colgroup>
+<col width="25%" />
+<col width="25%" />
+<col width="25%" />
+<col width="25%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>column</th>
+<th>type</th>
+<th>references</th>
+<th>description</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td><code class="ph codeph">filespace_oid</code></td>
+<td>oid</td>
+<td>pg_filespace.oid</td>
+<td>object id of the filespace</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">db_id</code></td>
+<td>smallint</td>
+<td> </td>
+<td>primary segment id</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">location</code></td>
+<td>text</td>
+<td> </td>
+<td>primary filesystem location</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">persistent_state</code></td>
+<td>smallint</td>
+<td> </td>
+<td>0 - free
+<p>1 - create pending</p>
+<p>2 - created</p>
+<p>3 - drop pending</p>
+<p>4 - aborting create</p>
+<p>5 - "Just in Time" create pending</p>
+<p>6 - bulk load create pending</p></td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">reserved</code></td>
+<td>integer</td>
+<td> </td>
+<td>Not used. Reserved for future use.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">parent_xid</code></td>
+<td>integer</td>
+<td> </td>
+<td>Global transaction id.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">persistent_serial_num</code></td>
+<td>bigint</td>
+<td> </td>
+<td>Log sequence number position in the transaction log for a file block.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">previous_free_tid</code></td>
+<td>tid</td>
+<td> </td>
+<td>Used by HAWQ to internally manage persistent representations of file system objects.</td>
+</tr>
+</tbody>
+</table>
+
+
diff --git a/reference/catalog/gp_persistent_relation_node.html.md.erb b/reference/catalog/gp_persistent_relation_node.html.md.erb
new file mode 100644
index 0000000..f141cf7
--- /dev/null
+++ b/reference/catalog/gp_persistent_relation_node.html.md.erb
@@ -0,0 +1,85 @@
+---
+title: gp_persistent_relation_node
+---
+
+The `gp_persistent_relation_node` table table keeps track of the status of file system objects in relation to the transaction status of relation objects (tables, view, indexes, and so on). This information is used to make sure the state of the system catalogs and the file system files persisted to disk are synchronized.
+
+<a id="topic1__fk138428"></a>
+<span class="tablecap">Table 1. pg\_catalog.gp\_persistent\_relation\_node</span>
+
+<table>
+<colgroup>
+<col width="25%" />
+<col width="25%" />
+<col width="25%" />
+<col width="25%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>column</th>
+<th>type</th>
+<th>references</th>
+<th>description</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td><code class="ph codeph">tablespace_oid</code></td>
+<td>oid</td>
+<td>pg_tablespace.oid</td>
+<td>Tablespace object id</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">database_oid</code></td>
+<td>oid</td>
+<td>pg_database.oid</td>
+<td>Database object id</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">relfilenode_oid</code></td>
+<td>oid</td>
+<td>pg_class.relfilenode</td>
+<td>The object id of the relation file node.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">persistent_state</code></td>
+<td>smallint</td>
+<td> </td>
+<td>0 - free
+<p>1 - create pending</p>
+<p>2 - created</p>
+<p>3 - drop pending</p>
+<p>4 - aborting create</p>
+<p>5 - "Just in Time" create pending</p>
+<p>6 - bulk load create pending</p></td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">reserved</code></td>
+<td>integer</td>
+<td> </td>
+<td>Unused. Reserved for future use.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">parent_xid</code></td>
+<td>integer</td>
+<td> </td>
+<td>Global transaction id.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">persistent_serial_num</code></td>
+<td>bigint</td>
+<td> </td>
+<td>Log sequence number position in the transaction log for a file block.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">previous_free_tid</code></td>
+<td>tid</td>
+<td> </td>
+<td>Used by HAWQ to internally manage persistent representations of file system objects.</td>
+</tr>
+</tbody>
+</table>
+
+
+
+
diff --git a/reference/catalog/gp_persistent_relfile_node.html.md.erb b/reference/catalog/gp_persistent_relfile_node.html.md.erb
new file mode 100644
index 0000000..6d24a41
--- /dev/null
+++ b/reference/catalog/gp_persistent_relfile_node.html.md.erb
@@ -0,0 +1,96 @@
+---
+title: gp_persistent_relfile_node
+---
+
+The `gp_persistent_relfile_node` table keeps track of the status of file system objects in relation to the transaction status of database objects. This information is used to make sure the state of the system catalogs and the file system files persisted to disk are synchronized.
+
+<a id="topic1__fk138428"></a>
+<span class="tablecap">Table 1. pg\_catalog.gp\_persistent\_relfile\_node</span>
+<table>
+<colgroup>
+<col width="25%" />
+<col width="25%" />
+<col width="25%" />
+<col width="25%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>column</th>
+<th>type</th>
+<th>references</th>
+<th>description</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td><code class="ph codeph">tablespace_oid</code></td>
+<td>oid</td>
+<td>pg_tablespace.oid</td>
+<td>Tablespace object id</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">database_oid</code></td>
+<td>oid</td>
+<td>pg_database.oid</td>
+<td>Database object id</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">relfilenode_oid</code></td>
+<td>oid</td>
+<td>pg_class.relfilenode</td>
+<td>The object id of the relation file node.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">segment_file_num</code></td>
+<td>integer</td>
+<td> </td>
+<td> </td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">relation_storage_manager</code></td>
+<td>smallint</td>
+<td> </td>
+<td> </td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">persistent_state</code></td>
+<td>smallint</td>
+<td> </td>
+<td>0 - free
+<p>1 - create pending</p>
+<p>2 - created</p>
+<p>3 - drop pending</p>
+<p>4 - aborting create</p>
+<p>5 - "Just in Time" create pending</p>
+<p>6 - bulk load create pending</p></td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">relation_bufpool_kind</code></td>
+<td>integer</td>
+<td> </td>
+<td> </td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">parent_xid</code></td>
+<td>integer</td>
+<td> </td>
+<td>Global transaction id.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">persistent_serial_num</code></td>
+<td>bigint</td>
+<td> </td>
+<td>Log sequence number position in the transaction log for a file block.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">previous_free_tid</code></td>
+<td>tid</td>
+<td> </td>
+<td>Used by HAWQ to internally manage persistent representations of file system objects.</td>
+</tr>
+</tbody>
+</table>
+
+
+
+
diff --git a/reference/catalog/gp_persistent_tablespace_node.html.md.erb b/reference/catalog/gp_persistent_tablespace_node.html.md.erb
new file mode 100644
index 0000000..55c853e
--- /dev/null
+++ b/reference/catalog/gp_persistent_tablespace_node.html.md.erb
@@ -0,0 +1,72 @@
+---
+title: gp_persistent_tablespace_node
+---
+
+The `gp_persistent_tablespace_node` table keeps track of the status of file system objects in relation to the transaction status of tablespace objects. This information is used to make sure the state of the system catalogs and the file system files persisted to disk are synchronized.
+
+<a id="topic1__fm138428"></a>
+<span class="tablecap">Table 1. pg\_catalog.gp\_persistent\_tablespace\_node</span>
+<table>
+<colgroup>
+<col width="25%" />
+<col width="25%" />
+<col width="25%" />
+<col width="25%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>column</th>
+<th>type</th>
+<th>references</th>
+<th>description</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td><code class="ph codeph">filespace_oid</code></td>
+<td>oid</td>
+<td>pg_filespace.oid</td>
+<td>Filespace object id</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">tablespace_oid</code></td>
+<td>oid</td>
+<td>pg_tablespace.oid</td>
+<td>Tablespace object id</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">persistent_state</code></td>
+<td>smallint</td>
+<td> </td>
+<td>0 - free
+<p>1 - create pending</p>
+<p>2 - created</p>
+<p>3 - drop pending</p>
+<p>4 - aborting create</p>
+<p>5 - "Just in Time" create pending</p>
+<p>6 - bulk load create pending</p></td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">parent_xid</code></td>
+<td>integer</td>
+<td> </td>
+<td>Global transaction id.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">persistent_serial_num</code></td>
+<td>bigint</td>
+<td> </td>
+<td>Log sequence number position in the transaction log for a file block.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">previous_free_tid</code></td>
+<td>tid</td>
+<td> </td>
+<td>Used by HAWQ to internally manage persistent representations of file system objects.</td>
+</tr>
+</tbody>
+</table>
+
+
+
+
diff --git a/reference/catalog/gp_relfile_node.html.md.erb b/reference/catalog/gp_relfile_node.html.md.erb
new file mode 100644
index 0000000..971ff16
--- /dev/null
+++ b/reference/catalog/gp_relfile_node.html.md.erb
@@ -0,0 +1,19 @@
+---
+title: gp_relfile_node
+---
+
+The `gp_relfile_node` table contains information about the file system objects for a relation (table, view, index, and so on).
+
+<a id="topic1__fo138428"></a>
+<span class="tablecap">Table 1. pg\_catalog.gp\_relation\_node</span>
+
+| column | type | references | description |
+|-------------------------|---------|-----------------------|--------------------------------------------------------------------------------------|
+| `relfilenode_oid` | oid | pg\_class.relfilenode | The object id of the relation file node. |
+| `segment_file_num` | integer | | For append-only tables, the append-only segment file number. |
+| `persistent_tid` | tid | | Used by HAWQ to internally manage persistent representations of file system objects. |
+| `persistent_serial_num` | bigint | | Log sequence number position in the transaction log for a file block. |
+
+
+
+
diff --git a/reference/catalog/gp_segment_configuration.html.md.erb b/reference/catalog/gp_segment_configuration.html.md.erb
new file mode 100644
index 0000000..d314515
--- /dev/null
+++ b/reference/catalog/gp_segment_configuration.html.md.erb
@@ -0,0 +1,25 @@
+---
+title: gp_segment_configuration
+---
+
+The `gp_segment_configuration` table contains information about master, standby and segment configuration.
+
+The HAWQ fault tolerance service (FTS) automatically detects the status of individual segments and marks the status of each segment in this table. If a segment is marked as DOWN, the corresponding reason is recorded in the [gp_configuration_history](gp_configuration_history.html) table. See [Understanding the Fault Tolerance Service](../../admin/FaultTolerance.html) for a description of the fault tolerance service.
+
+<a id="topic1__fr163962"></a>
+<span class="tablecap">Table 1. pg\_catalog.gp\_segment\_configuration</span>
+
+| column | type | references | description |
+|----------------------|---------|------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `registration_order` | integer | | When HAWQ starts, the master and each segment starts itself separately. This column indicates the order in which a segment node registers itself to master node. The registration\_order for segments starts from 1. Master's registration\_order is 0. Standby's registration\_order is -1. |
+| `role` | char | | The role that a node is currently running as. Values are `p` (segment), `m`(master) or `s `(standby). |
+| `status` | char | | The fault status of a segment. Values are `u` (up) or `d` (down). |
+| `port` | integer | | The TCP port the database server listener process is using. |
+| `hostname` | text | | The hostname of a segment host. |
+| `address` | text | | The hostname used to access a particular segment on a segment host. This value may be the same as `hostname` in systems upgraded from 3.x or on systems that do not have per-interface hostnames configured. |
+| `failed_tmpdir_num` | integer | | The number of failed temporary directories on the segment. User- configured temporary directories may fail on segments due to disk errors. This information is reported to the master. |
+| `failed_tmpdir` | text | | A list of failed temporary directories on the segment. Multiple failed temporary directories are separated by commas. |
+
+
+
+
diff --git a/reference/catalog/gp_version_at_initdb.html.md.erb b/reference/catalog/gp_version_at_initdb.html.md.erb
new file mode 100644
index 0000000..9e922e1
--- /dev/null
+++ b/reference/catalog/gp_version_at_initdb.html.md.erb
@@ -0,0 +1,17 @@
+---
+title: gp_version_at_initdb
+---
+
+The `gp_version_at_initdb` table is populated on the master and each segment in the HAWQ system. It identifies the version of HAWQ used when the system was first initialized. This table is defined in the `pg_global` tablespace, meaning it is globally shared across all databases in the system.
+
+<a id="topic1__ft142845"></a>
+<span class="tablecap">Table 1. pg\_catalog.gp\_version</span>
+
+| column | type | references | description |
+|------------------|---------|------------|-------------------------|
+| `schemaversion` | integer | | Schema version number. |
+| `productversion` | text | | Product version number. |
+
+
+
+
diff --git a/reference/catalog/pg_aggregate.html.md.erb b/reference/catalog/pg_aggregate.html.md.erb
new file mode 100644
index 0000000..1a02e89
--- /dev/null
+++ b/reference/catalog/pg_aggregate.html.md.erb
@@ -0,0 +1,25 @@
+---
+title: pg_aggregate
+---
+
+The `pg_aggregate` table stores information about aggregate functions. An aggregate function is a function that operates on a set of values (typically one column from each row that matches a query condition) and returns a single value computed from all these values. Typical aggregate functions are `sum`, `count`, and `max`. Each entry in `pg_aggregate` is an extension of an entry in `pg_proc`. The `pg_proc` entry carries the aggregate's name, input and output data types, and other information that is similar to ordinary functions.
+
+<a id="topic1__fu141982"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_aggregate</span>
+
+| column | type | references | description |
+|------------------|---------|------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `aggfnoid` | regproc | pg\_proc.oid | Aggregate function OID |
+| `aggtransfn` | regproc | pg\_proc.oid | Transition function OID |
+| `aggprelimfn` | regproc | | Preliminary function OID (zero if none) |
+| `aggfinalfn` | regproc | pg\_proc.oid | Final function OID (zero if none) |
+| `agginitval` | text | | The initial value of the transition state. This is a text field containing the initial value in its external string representation. If this field is NULL, the transition state value starts out NULL |
+| `agginvtransfn ` | regproc | pg\_proc.oid | The OID in pg\_proc of the inverse function of *aggtransfn* |
+| `agginvprelimfn` | regproc | pg\_proc.oid | The OID in pg\_proc of the inverse function of aggprelimfn |
+| `aggordered` | boolean | | If `true`, the aggregate is defined as `ORDERED`. |
+| `aggsortop` | oid | pg\_operator.oid | Associated sort operator OID (zero if none) |
+| `aggtranstype` | oid | pg\_type.oid | Data type of the aggregate function's internal transition (state) data |
+
+
+
+
diff --git a/reference/catalog/pg_am.html.md.erb b/reference/catalog/pg_am.html.md.erb
new file mode 100644
index 0000000..96ba56b
--- /dev/null
+++ b/reference/catalog/pg_am.html.md.erb
@@ -0,0 +1,38 @@
+---
+title: pg_am
+---
+
+The `pg_am` table stores information about index access methods. There is one row for each index access method supported by the system.
+
+<a id="topic1__fv141982"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_am</span>
+
+| column | type | references | description |
+|-------------------|----------|--------------|------------------------------------------------------------------------------------------------------------------------------|
+| `amname` | name | | Name of the access method |
+| `amstrategies` | smallint | | Number of operator strategies for this access method |
+| `amsupport` | smallint | | Number of support routines for this access method |
+| `amorderstrategy` | smallint | | Zero if the index offers no sort order, otherwise the strategy number of the strategy operator that describes the sort order |
+| `amcanunique` | boolean | | Does the access method support unique indexes? |
+| `amcanmulticol` | boolean | | Does the access method support multicolumn indexes? |
+| `amoptionalkey` | boolean | | Does the access method support a scan without any constraint for the first index column? |
+| `amindexnulls` | boolean | | Does the access method support null index entries? |
+| `amstorage` | boolean | | Can index storage data type differ from column data type? |
+| `amclusterable` | boolean | | Can an index of this type be clustered on? |
+| `aminsert` | regproc | pg\_proc.oid | "Insert this tuple" function |
+| `ambeginscan` | regproc | pg\_proc.oid | "Start new scan" function |
+| `amgettuple` | regproc | pg\_proc.oid | "Next valid tuple" function |
+| `amgetmulti` | regproc | pg\_proc.oid | "Fetch multiple tuples" function |
+| `amrescan` | regproc | pg\_proc.oid | "Restart this scan" function |
+| `amendscan` | regproc | pg\_proc.oid | "End this scan" function |
+| `ammarkpos` | regproc | pg\_proc.oid | "Mark current scan position" function |
+| `amrestrpos` | regproc | pg\_proc.oid | "Restore marked scan position" function |
+| `ambuild` | regproc | pg\_proc.oid | "Build new index" function |
+| `ambulkdelete` | regproc | pg\_proc.oid | Bulk-delete function |
+| `amvacuumcleanup` | regproc | pg\_proc.oid | Post-`VACUUM` cleanup function |
+| `amcostestimate` | regproc | pg\_proc.oid | Function to estimate cost of an index scan |
+| `amoptions` | regproc | pg\_proc.oid | Function to parse and validate reloptions for an index |
+
+
+
+
diff --git a/reference/catalog/pg_amop.html.md.erb b/reference/catalog/pg_amop.html.md.erb
new file mode 100644
index 0000000..c07bcd1
--- /dev/null
+++ b/reference/catalog/pg_amop.html.md.erb
@@ -0,0 +1,20 @@
+---
+title: pg_amop
+---
+
+The `pg_amop` table stores information about operators associated with index access method operator classes. There is one row for each operator that is a member of an operator class.
+
+<a id="topic1__fw143542"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_amop</span>
+
+| column | type | references | description |
+|----------------|----------|------------------|----------------------------------------------------------------------------|
+| `amopclaid` | oid | pg\_opclass.oid | The index operator class this entry is for |
+| `amopsubtype` | oid | pg\_type.oid | Subtype to distinguish multiple entries for one strategy; zero for default |
+| `amopstrategy` | smallint | | Operator strategy number |
+| `amopreqcheck` | boolean | | Index hit must be rechecked |
+| `amopopr` | oid | pg\_operator.oid | OID of the operator |
+
+
+
+
diff --git a/reference/catalog/pg_amproc.html.md.erb b/reference/catalog/pg_amproc.html.md.erb
new file mode 100644
index 0000000..7668bac
--- /dev/null
+++ b/reference/catalog/pg_amproc.html.md.erb
@@ -0,0 +1,19 @@
+---
+title: pg_amproc
+---
+
+The `pg_amproc` table stores information about support procedures associated with index access method operator classes. There is one row for each support procedure belonging to an operator class.
+
+<a id="topic1__fx143898"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_amproc</span>
+
+| column | type | references | description |
+|-----------------|---------|-----------------|--------------------------------------------|
+| `amopclaid` | oid | pg\_opclass.oid | The index operator class this entry is for |
+| `amprocsubtype` | oid | pg\_type.oid | Subtype, if cross-type routine, else zero |
+| `amprocnum` | int2 | | Support procedure number |
+| `amproc` | regproc | pg\_proc.oid | OID of the procedure |
+
+
+
+
diff --git a/reference/catalog/pg_appendonly.html.md.erb b/reference/catalog/pg_appendonly.html.md.erb
new file mode 100644
index 0000000..af243a8
--- /dev/null
+++ b/reference/catalog/pg_appendonly.html.md.erb
@@ -0,0 +1,29 @@
+---
+title: pg_appendonly
+---
+
+The `pg_appendonly` table contains information about the storage options and other characteristics of append-only tables.
+
+<a id="topic1__fy138428"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_appendonly</span>
+
+| column | type | references | description |
+|-----------------|----------|------------|----------------------------------------------------------------------------------------------------------------------------------------------------|
+| `relid` | oid | | The table object identifier (OID) of the compressed table. |
+| `compresslevel` | smallint | | The compression level, with compression ratio increasing from 1 to 9. If `zlib` is specified, valid values are also 1-9. |
+| `majorversion` | smallint | | The major version number of the pg\_appendonly table. |
+| `minorversion` | smallint | | The minor version number of the pg\_appendonly table. |
+| `checksum` | boolean | | A checksum value that is stored to compare the state of a block of data at compression time and at scan time to ensure data integrity. |
+| `compresstype` | text | | Type of compression used to compress append-only tables. The default is `zlib` (gzip compression). The `quicklz` compresstype has been deprecated. |
+| `columnstore` | boolean | | `0` for row-oriented storage. |
+| `segrelid` | oid | | Table on-disk segment file id. |
+| `segidxid` | oid | | Index on-disk segment file id. |
+| `blkdirrelid` | oid | | Block used for on-disk column-oriented table file. |
+| `blkdiridxid` | oid | | Block used for on-disk column-oriented index file. |
+| `version` | integer | | Version of MemTuples and block layout for this table. |
+| `pagesize` | integer | | The max page size of this relation. Only valid for Parquet tables; otherwise, the value is 0. |
+| `splitsize` | integer | | Size of a split. Default value is 64M, which is controlled by server configuration parameter `appendonly_split_write_size_mb`. |
+
+
+
+
diff --git a/reference/catalog/pg_attrdef.html.md.erb b/reference/catalog/pg_attrdef.html.md.erb
new file mode 100644
index 0000000..f4af77d
--- /dev/null
+++ b/reference/catalog/pg_attrdef.html.md.erb
@@ -0,0 +1,19 @@
+---
+title: pg_attrdef
+---
+
+The `pg_attrdef` table stores column default values. The main information about columns is stored in [pg\_attribute](pg_attribute.html#topic1). Only columns that explicitly specify a default value (when the table is created or the column is added) will have an entry here.
+
+<a id="topic1__fz143898"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_attrdef</span>
+
+| column | type | references | description |
+|-----------|----------|----------------------|-------------------------------------------------------------------------------------------------------|
+| `adrelid` | oid | pg\_class.oid | The table this column belongs to |
+| `adnum` | smallint | pg\_attribute.attnum | The number of the column |
+| `adbin ` | text | | The internal representation of the column default value |
+| `adsrc` | text | | A human-readable representation of the default value. This field is historical, and is best not used. |
+
+
+
+
diff --git a/reference/catalog/pg_attribute.html.md.erb b/reference/catalog/pg_attribute.html.md.erb
new file mode 100644
index 0000000..09f65bd
--- /dev/null
+++ b/reference/catalog/pg_attribute.html.md.erb
@@ -0,0 +1,32 @@
+---
+title: pg_attribute
+---
+
+The `pg_attribute` table stores information about table columns. There will be exactly one `pg_attribute` row for every column in every table in the database. (There will also be attribute entries for indexes, and all objects that have `pg_class` entries.) The term attribute is equivalent to column.
+
+<a id="topic1__ga143898"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_attribute</span>
+
+| column | type | references | description |
+|-----------------|----------|---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `attrelid` | oid | pg\_class.oid | The table this column belongs to |
+| `attname` | name | | The column name |
+| `atttypid` | oid | pg\_type.oid | The data type of this column |
+| `attstattarget` | integer | | Controls the level of detail of statistics accumulated for this column by `ANALYZE`. A zero value indicates that no statistics should be collected. A negative value says to use the system default statistics target. The exact meaning of positive values is data type-dependent. For scalar data types, it is both the target number of "most common values" to collect, and the target number of histogram bins to create. |
+| `attlen` | smallint | | A copy of pg\_type.typlen of this column's type. |
+| `attnum` | smallint | | The number of the column. Ordinary columns are numbered from 1 up. System columns, such as oid, have (arbitrary) negative numbers. |
+| `attndims` | integer | | Number of dimensions, if the column is an array type; otherwise `0`. (Presently, the number of dimensions of an array is not enforced, so any nonzero value effectively means it is an array) |
+| `attcacheoff` | integer | | Always `-1` in storage, but when loaded into a row descriptor in memory this may be updated to cache the offset of the attribute within the row |
+| `atttypmod` | integer | | Records type-specific data supplied at table creation time (for example, the maximum length of a varchar column). It is passed to type-specific input functions and length coercion functions. The value will generally be `-1` for types that do not need it. |
+| `attbyval` | boolean | | A copy of pg\_type.typbyval of this column's type |
+| `attstorage` | char | | Normally a copy of pg\_type.typstorage of this column's type. For TOAST-able data types, this can be altered after column creation to control storage policy. |
+| `attalign` | char | | A copy of pg\_type.typalign of this column's type |
+| `attnotnull` | boolean | | This represents a not-null constraint. It is possible to change this column to enable or disable the constraint. |
+| `atthasdef` | boolean | | This column has a default value, in which case there will be a corresponding entry in the pg\_attrdef catalog that actually defines the value |
+| `attisdropped` | boolean | | This column has been dropped and is no longer valid. A dropped column is still physically present in the table, but is ignored by the parser and so cannot be accessed via SQL |
+| `attislocal` | boolean | | This column is defined locally in the relation. Note that a column may be locally defined and inherited simultaneously |
+| `attinhcount` | integer | | The number of direct ancestors this column has. A column with a nonzero number of ancestors cannot be dropped nor renamed |
+
+
+
+
diff --git a/reference/catalog/pg_attribute_encoding.html.md.erb b/reference/catalog/pg_attribute_encoding.html.md.erb
new file mode 100644
index 0000000..3067a93
--- /dev/null
+++ b/reference/catalog/pg_attribute_encoding.html.md.erb
@@ -0,0 +1,18 @@
+---
+title: pg_attribute_encoding
+---
+
+The `pg_attribute_encoding` system catalog table contains column storage information.
+
+<a id="topic1__gb177839"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_attribute\_encoding</span>
+
+| column | type | modifers | storage | description |
+|--------------|------------|----------|----------|----------------------------------------|
+| `attrelid` | oid | not null | plain | Foreign key to `pg_attribute.attrelid` |
+| `attnum` | smallint | not null | plain | Foreign key to `pg_attribute.attnum` |
+| `attoptions` | text \[ \] | | extended | The options |
+
+
+
+
diff --git a/reference/catalog/pg_auth_members.html.md.erb b/reference/catalog/pg_auth_members.html.md.erb
new file mode 100644
index 0000000..7e770e0
--- /dev/null
+++ b/reference/catalog/pg_auth_members.html.md.erb
@@ -0,0 +1,19 @@
+---
+title: pg_auth_members
+---
+
+The `pg_auth_members` system catalog table shows the membership relations between roles. Any non-circular set of relationships is allowed. Because roles are system-wide, `pg_auth_members` is shared across all databases of a HAWQ system.
+
+<a id="topic1__gc143898"></a>
+ <span class="tablecap">Table 1. pg\_catalog.pg\_auth\_members</span>
+
+| column | type | references | description |
+|----------------|---------|----------------|----------------------------------------------------|
+| `roleid` | oid | pg\_authid.oid | ID of the parent-level (group) role |
+| `member` | oid | pg\_authid.oid | ID of a member role |
+| `grantor` | oid | pg\_authid.oid | ID of the role that granted this membership |
+| `admin_option` | boolean | | True if role member may grant membership to others |
+
+
+
+
diff --git a/reference/catalog/pg_authid.html.md.erb b/reference/catalog/pg_authid.html.md.erb
new file mode 100644
index 0000000..df24e7f
--- /dev/null
+++ b/reference/catalog/pg_authid.html.md.erb
@@ -0,0 +1,36 @@
+---
+title: pg_authid
+---
+
+The `pg_authid` table contains information about database authorization identifiers (roles). A role subsumes the concepts of users and groups. A user is a role with the `rolcanlogin` flag set. Any role (with or without `rolcanlogin`) may have other roles as members. See [pg\_auth\_members](pg_auth_members.html#topic1).
+
+Since this catalog contains passwords, it must not be publicly readable. [pg\_roles](pg_roles.html#topic1) is a publicly readable view on `pg_authid` that blanks out the password field.
+
+Because user identities are system-wide, `pg_authid` is shared across all databases in a HAWQ system: there is only one copy of `pg_authid` per system, not one per database.
+
+<a id="topic1__gd143898"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_authid</span>
+
+| column | type | references | description |
+|---------------------|-------------|------------|-----------------------------------------------------------------------------------------------------------------------|
+| `rolname` | name | | Role name |
+| `rolsuper` | boolean | | Role has superuser privileges |
+| `rolinherit` | boolean | | Role automatically inherits privileges of roles it is a member of |
+| `rolcreaterole` | boolean | | Role may create more roles |
+| `rolcreatedb` | boolean | | Role may create databases |
+| `rolcatupdate` | boolean | | Role may update system catalogs directly. (Even a superuser may not do this unless this column is true) |
+| `rolcanlogin` | boolean | | Role may log in. That is, this role can be given as the initial session authorization identifier |
+| `rolconnlimit` | int4 | | For roles that can log in, this sets maximum number of concurrent connections this role can make. `-1` means no limit |
+| `rolpassword` | text | | Password (possibly encrypted); NULL if none |
+| `rolvaliduntil` | timestamptz | | Password expiry time (only used for password authentication); NULL if no expiration |
+| `rolconfig` | text\[\] | | Session defaults for server configuration parameters |
+| `relresqueue` | oid | | Object ID of the associated resource queue ID in *pg\_resqueue* |
+| `rolcreaterextgpfd` | boolean | | Privilege to create read external tables with the `gpfdist` or `gpfdists` protocol |
+| `rolcreaterexhttp` | boolean | | Privilege to create read external tables with the `http` protocol |
+| `rolcreatewextgpfd` | boolean | | Privilege to create write external tables with the `gpfdist` or `gpfdists` protocol |
+| `rolcreaterexthdfs` | boolean | | Privilege to create read external tables with the `gphdfs` protocol. (`gphdfs` is deprecated.) |
+| `rolcreatewexthdfs` | boolean | | Privilege to create write external tables with the `gphdfs` protocol. (`gphdfs` is deprecated.) |
+
+
+
+
diff --git a/reference/catalog/pg_cast.html.md.erb b/reference/catalog/pg_cast.html.md.erb
new file mode 100644
index 0000000..513c7a3
--- /dev/null
+++ b/reference/catalog/pg_cast.html.md.erb
@@ -0,0 +1,23 @@
+---
+title: pg_cast
+---
+
+The `pg_cast` table stores data type conversion paths, both built-in paths and those defined with `CREATE CAST`. The cast functions listed in `pg_cast` must always take the cast source type as their first argument type, and return the cast destination type as their result type. A cast function can have up to three arguments. The second argument, if present, must be type `integer`; it receives the type modifier associated with the destination type, or `-1` if there is none. The third argument, if present, must be type `boolean`; it receives `true` if the cast is an explicit cast, `false` otherwise.
+
+It is legitimate to create a `pg_cast` entry in which the source and target types are the same, if the associated function takes more than one argument. Such entries represent 'length coercion functions' that coerce values of the type to be legal for a particular type modifier value. Note however that at present there is no support for associating non-default type modifiers with user-created data types, and so this facility is only of use for the small number of built-in types that have type modifier syntax built into the grammar.
+
+When a `pg_cast` entry has different source and target types and a function that takes more than one argument, it represents converting from one type to another and applying a length coercion in a single step. When no such entry is available, coercion to a type that uses a type modifier involves two steps, one to convert between data types and a second to apply the modifier.
+
+<a id="topic1__ge143898"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_cast</span>
+
+| column | type | references | description |
+|---------------|------|--------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `castsource` | oid | pg\_type.oid | OID of the source data type. |
+| `casttarget` | oid | pg\_type.oid | OID of the target data type. |
+| `castfunc` | oid | pg\_proc.oid | The OID of the function to use to perform this cast. Zero is stored if the data types are binary compatible (that is, no run-time operation is needed to perform the cast). |
+| `castcontext` | char | | Indicates what contexts the cast may be invoked in. `e` means only as an explicit cast (using `CAST` or `::` syntax). `a` means implicitly in assignment to a target column, as well as explicitly. `i` means implicitly in expressions, as well as the other cases*.* |
+
+
+
+
diff --git a/reference/catalog/pg_class.html.md.erb b/reference/catalog/pg_class.html.md.erb
new file mode 100644
index 0000000..5dff1f2
--- /dev/null
+++ b/reference/catalog/pg_class.html.md.erb
@@ -0,0 +1,213 @@
+---
+title: pg_class
+---
+
+The system catalog table `pg_class` catalogs tables and most everything else that has columns or is otherwise similar to a table (also known as *relations*). This includes indexes (see also [pg\_index](pg_index.html#topic1)), sequences, views, composite types, and TOAST tables. Not all columns are meaningful for all relation types.
+
+<a id="topic1__gf143898"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_class</span>
+
+<table>
+<colgroup>
+<col width="25%" />
+<col width="25%" />
+<col width="25%" />
+<col width="25%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>column</th>
+<th>type</th>
+<th>references</th>
+<th>description</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td><code class="ph codeph">relname</code></td>
+<td>name</td>
+<td> </td>
+<td>Name of the table, index, view, etc.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">relnamespace</code></td>
+<td>oid</td>
+<td>pg_namespace.oid</td>
+<td>The OID of the namespace (schema) that contains this relation</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">reltype</code></td>
+<td>oid</td>
+<td>pg_type.oid</td>
+<td>The OID of the data type that corresponds to this table's row type, if any (zero for indexes, which have no pg_type entry)</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">relowner</code></td>
+<td>oid</td>
+<td>pg_authid.oid</td>
+<td>Owner of the relation</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">relam</code></td>
+<td>oid</td>
+<td>pg_am.oid</td>
+<td>If this is an index, the access method used (B-tree, Bitmap, hash, etc.)</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">relfilenode</code></td>
+<td>oid</td>
+<td> </td>
+<td>Name of the on-disk file of this relation; <code class="ph codeph">0</code> if none.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">reltablespace</code></td>
+<td>oid</td>
+<td>pg_tablespace.oid</td>
+<td>The tablespace in which this relation is stored. If zero, the database's default tablespace is implied. (Not meaningful if the relation has no on-disk file.)</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">relpages</code></td>
+<td>integer</td>
+<td> </td>
+<td>Size of the on-disk representation of this table in pages (of 32K each). This is only an estimate used by the planner. It is updated by <code class="ph codeph">VACUUM</code>, <code class="ph codeph">ANALYZE</code>, and a few DDL commands.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">reltuples</code></td>
+<td>real</td>
+<td> </td>
+<td>Number of rows in the table. This is only an estimate used by the planner. It is updated by <code class="ph codeph">VACUUM</code>, <code class="ph codeph">ANALYZE</code>, and a few DDL commands.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">reltoastrelid</code></td>
+<td>oid</td>
+<td>pg_class.oid</td>
+<td>OID of the TOAST table associated with this table, <code class="ph codeph">0</code> if none. The TOAST table stores large attributes "out of line" in a secondary table.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">reltoastidxid</code></td>
+<td>oid</td>
+<td>pg_class.oid</td>
+<td>For a TOAST table, the OID of its index. <code class="ph codeph">0</code> if not a TOAST table.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">relaosegidxid</code></td>
+<td>oid</td>
+<td> </td>
+<td>Deprecated.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">relaosegrelid</code></td>
+<td>oid</td>
+<td> </td>
+<td>Deprecated.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">relhasindex </code></td>
+<td>boolean</td>
+<td> </td>
+<td>True if this is a table and it has (or recently had) any indexes. This is set by <code class="ph codeph">CREATE INDEX</code>, but not cleared immediately by <code class="ph codeph">DROP INDEX</code>. <code class="ph codeph">VACUUM</code> will clear if it finds the table has no indexes.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">relisshared</code></td>
+<td>boolean</td>
+<td> </td>
+<td>True if this table is shared across all databases in the system. Only certain system catalog tables are shared.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">relkind</code></td>
+<td>char</td>
+<td> </td>
+<td>The type of object
+<p><code class="ph codeph">r</code> = heap table, <code class="ph codeph">i</code> = index, <code class="ph codeph">S</code> = sequence, <code class="ph codeph">v</code> = view, <code class="ph codeph">c</code> = composite type, <code class="ph codeph">t</code> = TOAST value, <code class="ph codeph">c</code> = composite type, <code class="ph codeph">u</code> = uncataloged temporary heap table</p></td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">relstorage</code></td>
+<td>char</td>
+<td> </td>
+<td>The storage mode of a table
+<p><code class="ph codeph">h</code> = heap, <code class="ph codeph">v</code> = virtual, <code class="ph codeph">x</code>= external table.</p></td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">relnatts</code></td>
+<td>smallint</td>
+<td> </td>
+<td>Number of user columns in the relation (system columns not counted). There must be this many corresponding entries in pg_attribute.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">relchecks</code></td>
+<td>smallint</td>
+<td> </td>
+<td>Number of check constraints on the table.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">reltriggers</code></td>
+<td>smallint</td>
+<td> </td>
+<td>Number of triggers on the table.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">relukeys</code></td>
+<td>smallint</td>
+<td> </td>
+<td>Unused</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">relfkeys</code></td>
+<td>smallint</td>
+<td> </td>
+<td>Unused</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">relrefs</code></td>
+<td>smallint</td>
+<td> </td>
+<td>Unused</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">relhasoids</code></td>
+<td>boolean</td>
+<td> </td>
+<td>True if an OID is generated for each row of the relation.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">relhaspkey</code></td>
+<td>boolean</td>
+<td> </td>
+<td>True if the table once had a primary key.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">relhasrules</code></td>
+<td>boolean</td>
+<td> </td>
+<td>True if table has rules.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">relhassubclass</code></td>
+<td>boolean</td>
+<td> </td>
+<td>True if table has (or once had) any inheritance children.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">relfrozenxid</code></td>
+<td>xid</td>
+<td> </td>
+<td>All transaction IDs before this one have been replaced with a permanent (frozen) transaction ID in this table. This is used to track whether the table needs to be vacuumed in order to prevent transaction ID wraparound or to allow pg_clog to be shrunk. Zero (<code class="ph codeph">InvalidTransactionId</code>) if the relation is not a table.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">relacl</code></td>
+<td>aclitem[]</td>
+<td> </td>
+<td>Access privileges assigned by <code class="ph codeph">GRANT</code> and <code class="ph codeph">REVOKE</code>.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">reloptions</code></td>
+<td>text[]</td>
+<td> </td>
+<td>Access-method-specific options, as "keyword=value" strings.</td>
+</tr>
+</tbody>
+</table>
+
+
+
+
diff --git a/reference/catalog/pg_compression.html.md.erb b/reference/catalog/pg_compression.html.md.erb
new file mode 100644
index 0000000..3524af0
--- /dev/null
+++ b/reference/catalog/pg_compression.html.md.erb
@@ -0,0 +1,22 @@
+---
+title: pg_compression
+---
+
+The `pg_compression` system catalog table describes the compression methods available..
+
+<a id="topic1__gg143898"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_compression</span>
+
+| column | type | modifers | storage | description |
+|--------------------|---------|----------|---------|-----------------------------------|
+| `compname` | name | not null | plain | Name of the compression |
+| `compconstructor` | regproc | not null | plain | Name of compression constructor |
+| `compdestructor` | regproc | not null | plain | Name of compression destructor |
+| `compcompressor` | regproc | not null | plain | Name of the compressor |
+| `compdecompressor` | regproc | not null | plain | Name of the decompressor |
+| `compvalidator` | regproc | not null | plain | Name of the compression validator |
+| `compowner` | oid | not null | plain | oid from pg\_authid |
+
+
+
+
diff --git a/reference/catalog/pg_constraint.html.md.erb b/reference/catalog/pg_constraint.html.md.erb
new file mode 100644
index 0000000..0f591fd
--- /dev/null
+++ b/reference/catalog/pg_constraint.html.md.erb
@@ -0,0 +1,30 @@
+---
+title: pg_constraint
+---
+
+The `pg_constraint` system catalog table stores check and foreign key constraints on tables. Column constraints are not treated specially. Every column constraint is equivalent to some table constraint. Not-null constraints are represented in the [pg\_attribute](pg_attribute.html#topic1) catalog table. Check constraints on domains are stored here, too.
+
+<a id="topic1__gh143898"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_constraint</span>
+
+| column | type | references | description |
+|-----------------|--------------|----------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `conname` | name | | Constraint name |
+| `connamespace` | oid | pg\_namespace.oid | The OID of the namespace (schema) that contains this constraint. |
+| `contype ` | char | | `c` = check constraint, `f` = foreign key constraint. |
+| `condeferrable` | boolean | | Is the constraint deferrable? |
+| `condeferred ` | boolean | | Is the constraint deferred by default? |
+| `conrelid` | oid | pg\_class.oid | The table this constraint is on; 0 if not a table constraint. |
+| `contypid ` | oid | pg\_type.oid | The domain this constraint is on; 0 if not a domain constraint. |
+| `confrelid` | oid | pg\_class.oid | If a foreign key, the referenced table; else 0. |
+| `confupdtype` | char | | Foreign key update action code. |
+| `confdeltype` | char | | Foreign key deletion action code. |
+| `confmatchtype` | char | | Foreign key match type. |
+| `conkey` | smallint\[\] | pg\_attribute.attnum | If a table constraint, list of columns which the constraint constrains. |
+| `confkey` | smallint\[\] | pg\_attribute.attnum | If a foreign key, list of the referenced columns. |
+| `conbin` | text | | If a check constraint, an internal representation of the expression. |
+| `consrc` | text | | If a check constraint, a human-readable representation of the expression. This is not updated when referenced objects change; for example, it won't track renaming of columns. Rather than relying on this field, it is best to use `pg_get_constraintdef()` to extract the definition of a check constraint. |
+
+
+
+
diff --git a/reference/catalog/pg_conversion.html.md.erb b/reference/catalog/pg_conversion.html.md.erb
new file mode 100644
index 0000000..43763bc
--- /dev/null
+++ b/reference/catalog/pg_conversion.html.md.erb
@@ -0,0 +1,22 @@
+---
+title: pg_conversion
+---
+
+The `pg_conversion` system catalog table describes the available encoding conversion procedures as defined by `CREATE CONVERSION`.
+
+<a id="topic1__gi143898"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_conversion</span>
+
+| column | type | references | description |
+|------------------|---------|-------------------|------------------------------------------------------------------|
+| `conname` | name | | Conversion name (unique within a namespace). |
+| `connamespace` | oid | pg\_namespace.oid | The OID of the namespace (schema) that contains this conversion. |
+| `conowner` | oid | pg\_authid.oid | Owner of the conversion. |
+| `conforencoding` | integer | | Source encoding ID. |
+| `contoencoding` | integer | | Destination encoding ID. |
+| `conproc` | regproc | pg\_proc.oid | Conversion procedure. |
+| `condefault` | boolean | | True if this is the default conversion. |
+
+
+
+
diff --git a/reference/catalog/pg_database.html.md.erb b/reference/catalog/pg_database.html.md.erb
new file mode 100644
index 0000000..b02a532
--- /dev/null
+++ b/reference/catalog/pg_database.html.md.erb
@@ -0,0 +1,26 @@
+---
+title: pg_database
+---
+
+The `pg_database` system catalog table stores information about the available databases. Databases are created with the `CREATE DATABASE` SQL command. Unlike most system catalogs, `pg_database` is shared across all databases in the system. There is only one copy of `pg_database` per system, not one per database.
+
+<a id="topic1__gj143898"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_database</span>
+
+| column | type | references | description |
+|-----------------|-------------|--------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `datname` | name | | Database name. |
+| `datdba` | oid | pg\_authid.oid | Owner of the database, usually the user who created it. |
+| `encoding` | integer | | Character encoding for this database. `pg_encoding_to_char()` can translate this number to the encoding name. |
+| `datistemplate` | boolean | | If true then this database can be used in the `TEMPLATE` clause of `CREATE DATABASE` to create a new database as a clone of this one. |
+| `datallowconn` | boolean | | If false then no one can connect to this database. This is used to protect the `template0` database from being altered. |
+| `datconnlimit` | integer | | Sets the maximum number of concurrent connections that can be made to this database. `-1` means no limit. |
+| `datlastsysoid` | oid | | Last system OID in the database. |
+| `datfrozenxid ` | xid | | All transaction IDs before this one have been replaced with a permanent (frozen) transaction ID in this database. This is used to track whether the database needs to be vacuumed in order to prevent transaction ID wraparound or to allow pg\_clog to be shrunk. It is the minimum of the per-table *pg\_class.relfrozenxid* values. |
+| `dattablespace` | oid | pg\_tablespace.oid | The default tablespace for the database. Within this database, all tables for which *pg\_class.reltablespace* is zero will be stored in this tablespace. All non-shared system catalogs will also be there. |
+| `datconfig` | text\[\] | | Session defaults for user-settable server configuration parameters. |
+| `datacl` | aclitem\[\] | | Database access privileges as given by `GRANT` and `REVOKE`. |
+
+
+
+
diff --git a/reference/catalog/pg_depend.html.md.erb b/reference/catalog/pg_depend.html.md.erb
new file mode 100644
index 0000000..85a1835
--- /dev/null
+++ b/reference/catalog/pg_depend.html.md.erb
@@ -0,0 +1,26 @@
+---
+title: pg_depend
+---
+
+The `pg_depend` system catalog table records the dependency relationships between database objects. This information allows `DROP` commands to find which other objects must be dropped by `DROP CASCADE` or prevent dropping in the `DROP RESTRICT` case. See also [pg\_shdepend](pg_shdepend.html#topic1), which performs a similar function for dependencies involving objects that are shared across a HAWQ system.
+
+In all cases, a `pg_depend` entry indicates that the referenced object may not be dropped without also dropping the dependent object. However, there are several subflavors identified by `deptype`:
+
+- **DEPENDENCY\_NORMAL (n)** — A normal relationship between separately-created objects. The dependent object may be dropped without affecting the referenced object. The referenced object may only be dropped by specifying `CASCADE`, in which case the dependent object is dropped, too. Example: a table column has a normal dependency on its data type.
+- **DEPENDENCY\_AUTO (a)** — The dependent object can be dropped separately from the referenced object, and should be automatically dropped (regardless of `RESTRICT` or `CASCADE` mode) if the referenced object is dropped. Example: a named constraint on a table is made autodependent on the table, so that it will go away if the table is dropped.
+- **DEPENDENCY\_INTERNAL (i)** — The dependent object was created as part of creation of the referenced object, and is really just a part of its internal implementation. A `DROP` of the dependent object will be disallowed outright (we'll tell the user to issue a `DROP` against the referenced object, instead). A `DROP` of the referenced object will be propagated through to drop the dependent object whether `CASCADE` is specified or not.
+- **DEPENDENCY\_PIN (p)** — There is no dependent object; this type of entry is a signal that the system itself depends on the referenced object, and so that object must never be deleted. Entries of this type are created only by system initialization. The columns for the dependent object contain zeroes. <a id="topic1__gk143898"></a>
+
+<span class="tablecap">Table 1. pg\_catalog.pg\_depend</span>
+
+ | column | type | references | description |
+ |----------------|---------|----------------|------------------------------------------------------------------------------------------------------------|
+ | `classid` | oid | pg\_class.oid | The OID of the system catalog the dependent object is in. |
+ | `objid` | oid | any OID column | The OID of the specific dependent object. |
+ | `objsubid ` | integer | | For a table column, this is the column number. For all other object types, this column is zero. |
+ | `refclassid` | oid | pg\_class.oid | The OID of the system catalog the referenced object is in. |
+ | `refobjid` | oid | any OID column | The OID of the specific referenced object. |
+ | `refobjsubid ` | integer | | For a table column, this is the referenced column number. For all other object types, this column is zero. |
+ | `deptype` | char | | A code defining the specific semantics of this dependency relationship. |
+
+
diff --git a/reference/catalog/pg_description.html.md.erb b/reference/catalog/pg_description.html.md.erb
new file mode 100644
index 0000000..bad9627
--- /dev/null
+++ b/reference/catalog/pg_description.html.md.erb
@@ -0,0 +1,17 @@
+---
+title: pg_description
+---
+
+The `pg_description` system catalog table stores optional descriptions (comments) for each database object. Descriptions can be manipulated with the `COMMENT` command and viewed with `psql`'s `\d` meta-commands. Descriptions of many built-in system objects are provided in the initial contents of `pg_description`. See also [pg\_shdescription](pg_shdescription.html#topic1), which performs a similar function for descriptions involving objects that are shared across a HAWQ system.
+
+<a id="topic1__gm143898"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_description</span>
+
+| column | type | references | description |
+|----------------|---------|----------------|--------------------------------------------------------------------------------------------------------------|
+| `objoid` | oid | any OID column | The OID of the object this description pertains to. |
+| `classoid` | oid | pg\_class.oid | The OID of the system catalog this object appears in |
+| `objsubid ` | integer | | For a comment on a table column, this is the column number. For all other object types, this column is zero. |
+| `description ` | text | | Arbitrary text that serves as the description of this object. |
+
+
diff --git a/reference/catalog/pg_exttable.html.md.erb b/reference/catalog/pg_exttable.html.md.erb
new file mode 100644
index 0000000..ca5fc88
--- /dev/null
+++ b/reference/catalog/pg_exttable.html.md.erb
@@ -0,0 +1,23 @@
+---
+title: pg_exttable
+---
+
+The `pg_exttable` system catalog table is used to track external tables and web tables created by the `CREATE EXTERNAL TABLE` command.
+
+<a id="topic1__gn143898"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_exttable</span>
+
+| column | type | references | description |
+|-------------------|----------|---------------|------------------------------------------------------------------------------------------------------------------|
+| `reloid` | oid | pg\_class.oid | The OID of this external table. |
+| `location` | text\[\] | | The URI location(s) of the external table files. |
+| `fmttype` | char | | Format of the external table files: `t` for text, or `c` for csv. |
+| `fmtopts` | text | | Formatting options of the external table files, such as the field delimiter, null string, escape character, etc. |
+| `command` | text | | The OS command to execute when the external table is accessed. |
+| `rejectlimit` | integer | | The per segment reject limit for rows with errors, after which the load will fail. |
+| `rejectlimittype` | char | | Type of reject limit threshold: `r` for number of rows. |
+| `fmterrtbl` | oid | pg\_class.oid | The object id of the error table where format errors will be logged. |
+| `encoding` | text | | The client encoding. |
+| `writable` | boolean | | `0` for readable external tables, `1` for writable external tables. |
+
+
diff --git a/reference/catalog/pg_filespace.html.md.erb b/reference/catalog/pg_filespace.html.md.erb
new file mode 100644
index 0000000..e0b810e
--- /dev/null
+++ b/reference/catalog/pg_filespace.html.md.erb
@@ -0,0 +1,19 @@
+---
+title: pg_filespace
+---
+
+The `pg_filespace` table contains information about the filespaces created in a HAWQ system. Every system contains a default filespace, `pg_system`, which is a collection of all the data directory locations created at system initialization time.
+
+A tablespace requires a file system location to store its database files. In HAWQ, the master and each segment needs its own distinct storage location. This collection of file system locations for all components in a HAWQ system is referred to as a filespace.
+
+<a id="topic1__go138428"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_filespace</span>
+
+| column | type | references | description |
+|-----------|------|---------------|-------------------------------------------------------|
+| `fsname` | name | | The name of the filespace. |
+| `fsowner` | oid | pg\_roles.oid | The object id of the role that created the filespace. |
+
+
+
+
diff --git a/reference/catalog/pg_filespace_entry.html.md.erb b/reference/catalog/pg_filespace_entry.html.md.erb
new file mode 100644
index 0000000..5a45113
--- /dev/null
+++ b/reference/catalog/pg_filespace_entry.html.md.erb
@@ -0,0 +1,18 @@
+---
+title: pg_filespace_entry
+---
+
+A tablespace requires a file system location to store its database files. In HAWQ, the master and each segment needs its own distinct storage location. This collection of file system locations for all components in a HAWQ system is referred to as a *filespace*. The `pg_filespace_entry` table contains information about the collection of file system locations across a HAWQ system that comprise a HAWQ filespace.
+
+<a id="topic1__gp138428"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_filespace\_entry</span>
+
+
+| column | type | references | description |
+|---------------|---------|----------------------------------|-------------------------------------------|
+| `fsefsoid` | oid | pg\_filespace.oid | Object id of the filespace. |
+| `fsedbid` | integer | gp\_segment\_ configuration.dbid | Segment id. |
+| `fselocation` | text | | File system location for this segment id. |
+
+
+
diff --git a/reference/catalog/pg_index.html.md.erb b/reference/catalog/pg_index.html.md.erb
new file mode 100644
index 0000000..cd1a054
--- /dev/null
+++ b/reference/catalog/pg_index.html.md.erb
@@ -0,0 +1,23 @@
+---
+title: pg_index
+---
+
+The `pg_index` system catalog table contains part of the information about indexes. The rest is mostly in [pg\_class](pg_class.html#topic1).
+
+<a id="topic1__gq143898"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_index</span>
+
+| column | type | references | description |
+|------------------|------------|----------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `indexrelid` | oid | pg\_class.oid | The OID of the pg\_class entry for this index. |
+| `indrelid` | oid | pg\_class.oid | The OID of the pg\_class entry for the table this index is for. |
+| `indnatts` | smallint | | The number of columns in the index (duplicates pg\_class.relnatts). |
+| `indisunique` | boolean | | If true, this is a unique index. |
+| `indisclustered` | boolean | | If true, the table was last clustered on this index via the `CLUSTER` command. |
+| `indisvalid` | boolean | | If true, the index is currently valid for queries. False means the index is possibly incomplete: it must still be modified by `INSERT`/`UPDATE` operations, but it cannot safely be used for queries. |
+| `indkey` | int2vector | pg\_attribute.attnum | This is an array of indnatts values that indicate which table columns this index indexes. For example a value of 1 3 would mean that the first and the third table columns make up the index key. A zero in this array indicates that the corresponding index attribute is an expression over the table columns, rather than a simple column reference. |
+| `indclass` | oidvector | pg\_opclass.oid | For each column in the index key this contains the OID of the operator class to use. |
+| `indexprs` | text | | Expression trees (in `nodeToString()` representation) for index attributes that are not simple column references. This is a list with one element for each zero entry in indkey. NULL if all index attributes are simple references. |
+| `indpred` | text | | Expression tree (in `nodeToString()` representation) for partial index predicate. NULL if not a partial index. |
+
+
diff --git a/reference/catalog/pg_inherits.html.md.erb b/reference/catalog/pg_inherits.html.md.erb
new file mode 100644
index 0000000..9868602
--- /dev/null
+++ b/reference/catalog/pg_inherits.html.md.erb
@@ -0,0 +1,16 @@
+---
+title: pg_inherits
+---
+
+The `pg_inherits` system catalog table records information about table inheritance hierarchies. There is one entry for each direct child table in the database. (Indirect inheritance can be determined by following chains of entries.) In HAWQ, inheritance relationships are created by both the `INHERITS` clause (standalone inheritance) and the `PARTITION BY` clause (partitioned child table inheritance) of `CREATE TABLE`.
+
+<a id="topic1__gr143898"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_inherits</span>
+
+| column | type | references | description |
+|-------------|---------|---------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `inhrelid` | oid | pg\_class.oid | The OID of the child table. |
+| `inhparent` | oid | pg\_class.oid | The OID of the parent table. |
+| `inhseqno` | integer | | If there is more than one direct parent for a child table (multiple inheritance), this number tells the order in which the inherited columns are to be arranged. The count starts at 1. |
+
+
diff --git a/reference/catalog/pg_language.html.md.erb b/reference/catalog/pg_language.html.md.erb
new file mode 100644
index 0000000..9b626f9
--- /dev/null
+++ b/reference/catalog/pg_language.html.md.erb
@@ -0,0 +1,21 @@
+---
+title: pg_language
+---
+
+The `pg_language` system catalog table registers languages in which you can write functions or stored procedures. It is populated by `CREATE LANGUAGE`.
+
+<a id="topic1__gs143898"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_language</span>
+
+| column | type | references | description |
+|-----------------|-------------|--------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `lanname` | name | | Name of the language. |
+| `lanispl ` | boolean | | This is false for internal languages (such as SQL) and true for user-defined languages. Currently, `pg_dump` still uses this to determine which languages need to be dumped, but this may be replaced by a different mechanism in the future. |
+| `lanpltrusted ` | boolean | | True if this is a trusted language, which means that it is believed not to grant access to anything outside the normal SQL execution environment. Only superusers may create functions in untrusted languages. |
+| `lanplcallfoid` | oid | pg\_proc.oid | For noninternal languages this references the language handler, which is a special function that is responsible for executing all functions that are written in the particular language. |
+| `lanvalidator` | oid | pg\_proc.oid | This references a language validator function that is responsible for checking the syntax and validity of new functions when they are created. Zero if no validator is provided. |
+| `lanacl ` | aclitem\[\] | | Access privileges for the language. |
+
+
+
+
diff --git a/reference/catalog/pg_largeobject.html.md.erb b/reference/catalog/pg_largeobject.html.md.erb
new file mode 100644
index 0000000..59d2c6d
--- /dev/null
+++ b/reference/catalog/pg_largeobject.html.md.erb
@@ -0,0 +1,19 @@
+---
+title: pg_largeobject
+---
+
+The `pg_largeobject` system catalog table holds the data making up 'large objects'. A large object is identified by an OID assigned when it is created. Each large object is broken into segments or 'pages' small enough to be conveniently stored as rows in `pg_largeobject`. The amount of data per page is defined to be `LOBLKSIZE` (which is currently `BLCKSZ`/4, or typically 8K).
+
+Each row of `pg_largeobject` holds data for one page of a large object, beginning at byte offset (*pageno*` * LOBLKSIZE`) within the object. The implementation allows sparse storage: pages may be missing, and may be shorter than `LOBLKSIZE` bytes even if they are not the last page of the object. Missing regions within a large object read as zeroes.
+
+<a id="topic1__gt143898"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_largeobject</span>
+
+| column | type | references | description |
+|----------|---------|------------|---------------------------------------------------------------------------------------------------------|
+| `loid` | oid | | Identifier of the large object that includes this page. |
+| `pageno` | integer | | Page number of this page within its large object (counting from zero). |
+| `data` | bytea | | Actual data stored in the large object. This will never be more than `LOBLKSIZE` bytes and may be less. |
+
+
+
diff --git a/reference/catalog/pg_listener.html.md.erb b/reference/catalog/pg_listener.html.md.erb
new file mode 100644
index 0000000..4df5fda
--- /dev/null
+++ b/reference/catalog/pg_listener.html.md.erb
@@ -0,0 +1,20 @@
+---
+title: pg_listener
+---
+
+The `pg_listener` system catalog table supports the `LISTEN` and `NOTIFY` commands. A listener creates an entry in `pg_listener` for each notification name it is listening for. A notifier scans and updates each matching entry to show that a notification has occurred. The notifier also sends a signal (using the PID recorded in the table) to awaken the listener from sleep.
+
+This table is not currently used in HAWQ.
+
+<a id="topic1__gu143898"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_listener</span>
+
+| column | type | references | description |
+|----------------|---------|------------|----------------------------------------------------------------------------------------------------------------------------------|
+| `relname` | name | | Notify condition name. (The name need not match any actual relation in the database. |
+| `listenerpid` | integer | | PID of the server process that created this entry. |
+| `notification` | integer | | Zero if no event is pending for this listener. If an event is pending, the PID of the server process that sent the notification. |
+
+
+
+
diff --git a/reference/catalog/pg_locks.html.md.erb b/reference/catalog/pg_locks.html.md.erb
new file mode 100644
index 0000000..73a8712
--- /dev/null
+++ b/reference/catalog/pg_locks.html.md.erb
@@ -0,0 +1,35 @@
+---
+title: pg_locks
+---
+
+The `pg_locks` view provides access to information about the locks held by open transactions within HAWQ.
+
+`pg_locks` contains one row per active lockable object, requested lock mode, and relevant transaction. Thus, the same lockable object may appear many times, if multiple transactions are holding or waiting for locks on it. However, an object that currently has no locks on it will not appear at all.
+
+There are several distinct types of lockable objects: whole relations (such as tables), individual pages of relations, individual tuples of relations, transaction IDs, and general database objects. Also, the right to extend a relation is represented as a separate lockable object.
+
+<a id="topic1__gv141982"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_locks</span>
+
+| column | type | references | description |
+|------------------|----------|------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `locktype` | text | | Type of the lockable object: `relation`, `extend`, `page`, `tuple`, `transactionid`, `object`, `userlock`, `resource queue`, or `advisory` |
+| `database` | oid | pg\_database.oid | OID of the database in which the object exists, zero if the object is a shared object, or NULL if the object is a transaction ID |
+| `relation` | oid | pg\_class.oid | OID of the relation, or NULL if the object is not a relation or part of a relation |
+| `page ` | integer | | Page number within the relation, or NULL if the object is not a tuple or relation page |
+| `tuple ` | smallint | | Tuple number within the page, or NULL if the object is not a tuple |
+| `transactionid ` | xid | | ID of a transaction, or NULL if the object is not a transaction ID |
+| `classid` | oid | pg\_class.oid | OID of the system catalog containing the object, or NULL if the object is not a general database object |
+| `objid ` | oid | any OID column | OID of the object within its system catalog, or NULL if the object is not a general database object |
+| `objsubid ` | smallint | | For a table column, this is the column number (the classid and objid refer to the table itself). For all other object types, this column is zero. NULL if the object is not a general database object |
+| `transaction` | xid | | ID of the transaction that is holding or awaiting this lock |
+| `pid` | integer | | Process ID of the server process holding or awaiting this lock. NULL if the lock is held by a prepared transaction |
+| `mode` | text | | Name of the lock mode held or desired by this process |
+| `granted` | boolean | | True if lock is held, false if lock is awaited. |
+| `mppsessionid` | integer | | The id of the client session associated with this lock. |
+| `mppiswriter` | boolean | | Specifies whether the lock is held by a writer process. |
+| `gp_segment_id` | integer | | The HAWQ segment id (`dbid`) where the lock is held. |
+
+
+
+
diff --git a/reference/catalog/pg_namespace.html.md.erb b/reference/catalog/pg_namespace.html.md.erb
new file mode 100644
index 0000000..b307ecb
--- /dev/null
+++ b/reference/catalog/pg_namespace.html.md.erb
@@ -0,0 +1,18 @@
+---
+title: pg_namespace
+---
+
+The `pg_namespace` system catalog table stores namespaces. A namespace is the structure underlying SQL schemas: each namespace can have a separate collection of relations, types, etc. without name conflicts.
+
+<a id="topic1__gx143898"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_namespace</span>
+
+| column | type | references | description |
+|------------|-------------|----------------|-----------------------------------------------------|
+| `nspname` | name | | Name of the namespace |
+| `nspowner` | oid | pg\_authid.oid | Owner of the namespace |
+| `nspacl ` | aclitem\[\] | | Access privileges as given by `GRANT` and `REVOKE`. |
+
+
+
+
diff --git a/reference/catalog/pg_opclass.html.md.erb b/reference/catalog/pg_opclass.html.md.erb
new file mode 100644
index 0000000..d03315c
--- /dev/null
+++ b/reference/catalog/pg_opclass.html.md.erb
@@ -0,0 +1,22 @@
+---
+title: pg_opclass
+---
+
+The `pg_opclass` system catalog table defines index access method operator classes. Each operator class defines semantics for index columns of a particular data type and a particular index access method. Note that there can be multiple operator classes for a given data type/access method combination, thus supporting multiple behaviors. The majority of the information defining an operator class is actually not in its `pg_opclass` row, but in the associated rows in [pg\_amop](pg_amop.html#topic1) and [pg\_amproc](pg_amproc.html#topic1). Those rows are considered to be part of the operator class definition — this is not unlike the way that a relation is defined by a single [pg\_class](pg_class.html#topic1) row plus associated rows in [pg\_attribute](pg_attribute.html#topic1) and other tables.
+
+<a id="topic1__gw141982"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_opclass</span>
+
+| column | type | references | description |
+|----------------|---------|-------------------|-------------------------------------------------------------------------|
+| `opcamid` | oid | pg\_am.oid | Index access method operator class is for. |
+| `opcname` | name | | Name of this operator class |
+| `opcnamespace` | oid | pg\_namespace.oid | Namespace of this operator class |
+| `opcowner` | oid | pg\_authid.oid | Owner of the operator class |
+| `opcintype` | oid | pg\_type.oid | Data type that the operator class indexes. |
+| `opcdefault` | boolean | | True if this operator class is the default for the data type opcintype. |
+| `opckeytype` | oid | pg\_type.oid | Type of data stored in index, or zero if same as opcintype. |
+
+
+
+
diff --git a/reference/catalog/pg_operator.html.md.erb b/reference/catalog/pg_operator.html.md.erb
new file mode 100644
index 0000000..71c632d
--- /dev/null
+++ b/reference/catalog/pg_operator.html.md.erb
@@ -0,0 +1,32 @@
+---
+title: pg_operator
+---
+
+The `pg_operator` system catalog table stores information about operators, both built-in and those defined by `CREATE OPERATOR`. Unused column contain zeroes. For example, `oprleft` is zero for a prefix operator.
+
+<a id="topic1__gy150092"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_operator</span>
+
+| column | type | references | description |
+|----------------|---------|-------------------|--------------------------------------------------------------------------------------------------------------------------|
+| `oprname` | name | | Name of the operator. |
+| `oprnamespace` | oid | pg\_namespace.oid | The OID of the namespace that contains this operator. |
+| `oprowner` | oid | pg\_authid.oid | Owner of the operator. |
+| `oprkind` | char | | `b` = infix (both), `l` = prefix (left), `r` = postfix (right) |
+| `oprcanhash` | boolean | | This operator supports hash joins. |
+| `oprleft` | oid | pg\_type.oid | Type of the left operand. |
+| `oprright` | oid | pg\_type.oid | Type of the right operand. |
+| `oprresult` | oid | pg\_type.oid | Type of the result. |
+| `oprcom` | oid | pg\_operator.oid | Commutator of this operator, if any. |
+| `oprnegate` | | pg\_operator.oid | Negator of this operator, if any. |
+| `oprlsortop` | oid | pg\_operator.oid | If this operator supports merge joins, the operator that sorts the type of the left-hand operand (`L<L`). |
+| `oprrsortop` | oid | pg\_operator.oid | If this operator supports merge joins, the operator that sorts the type of the right-hand operand (`R<R`). |
+| `oprltcmpop` | oid | pg\_operator.oid | If this operator supports merge joins, the less-than operator that compares the left and right operand types (`L<R`). |
+| `oprgtcmpop` | oid | pg\_operator.oid | If this operator supports merge joins, the greater-than operator that compares the left and right operand types (`L>R`). |
+| `oprcode` | regproc | pg\_proc.oid | Function that implements this operator. |
+| `oprrest ` | regproc | pg\_proc.oid | Restriction selectivity estimation function for this operator. |
+| `oprjoin` | regproc | pg\_proc.oid | Join selectivity estimation function for this operator. |
+
+
+
+
diff --git a/reference/catalog/pg_partition.html.md.erb b/reference/catalog/pg_partition.html.md.erb
new file mode 100644
index 0000000..6795930
--- /dev/null
+++ b/reference/catalog/pg_partition.html.md.erb
@@ -0,0 +1,20 @@
+---
+title: pg_partition
+---
+
+The `pg_partition` system catalog table is used to track partitioned tables and their inheritance level relationships. Each row of `pg_partition` represents either the level of a partitioned table in the partition hierarchy, or a subpartition template description. The value of the attribute `paristemplate` determines what a particular row represents.
+
+<a id="topic1__gz143898"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_partition</span>
+
+| column | type | references | description |
+|-----------------|------------|-----------------|-----------------------------------------------------------------------------------------------------------------------------------------------------|
+| `parrelid` | oid | pg\_class.oid | The object identifier of the table. |
+| `parkind` | char | | The partition type - `R` for range or `L` for list. |
+| `parlevel` | smallint | | The partition level of this row: 0 for the top-level parent table, 1 for the first level under the parent table, 2 for the second level, and so on. |
+| `paristemplate` | boolean | | Whether or not this row represents a subpartition template definition (true) or an actual partitioning level (false). |
+| `parnatts` | smallint | | The number of attributes that define this level. |
+| `paratts` | int2vector | | An array of the attribute numbers (as in `pg_attribute.attnum`) of the attributes that participate in defining this level. |
+| `parclass` | oidvector | pg\_opclass.oid | The operator class identifier(s) of the partition columns. |
+
+
diff --git a/reference/catalog/pg_partition_columns.html.md.erb b/reference/catalog/pg_partition_columns.html.md.erb
new file mode 100644
index 0000000..2205a24
--- /dev/null
+++ b/reference/catalog/pg_partition_columns.html.md.erb
@@ -0,0 +1,20 @@
+---
+title: pg_partition_columns
+---
+
+The `pg_partition_columns` system view is used to show the partition key columns of a partitioned table.
+
+<a id="topic1__ha179967"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_partition\_columns</span>
+
+| column | type | references | description |
+|-----------------------------|----------|------------|--------------------------------------------------------------------------------------------------------------------------------------|
+| `schemaname` | name | | The name of the schema the partitioned table is in. |
+| `tablename` | name | | The table name of the top-level parent table. |
+| `columnname` | name | | The name of the partition key column. |
+| `partitionlevel` | smallint | | The level of this subpartition in the hierarchy. |
+| `position_in_partition_key` | integer | | For list partitions you can have a composite (multi-column) partition key. This shows the position of the column in a composite key. |
+
+
+
+
diff --git a/reference/catalog/pg_partition_encoding.html.md.erb b/reference/catalog/pg_partition_encoding.html.md.erb
new file mode 100644
index 0000000..e1dbabb
--- /dev/null
+++ b/reference/catalog/pg_partition_encoding.html.md.erb
@@ -0,0 +1,18 @@
+---
+title: pg_partition_encoding
+---
+
+The `pg_partition_encoding` system catalog table describes the available column compression options for a partition template.
+
+<a id="topic1__hb177831"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_attribute\_encoding</span>
+
+| column | type | modifers | storage | description |
+|--------------------|------------|----------|----------|-------------|
+| `parencoid` | oid | not null | plain | |
+| `parencattnum` | smallint | not null | plain | |
+| `parencattoptions` | text \[ \] | | extended | |
+
+
+
+
diff --git a/reference/catalog/pg_partition_rule.html.md.erb b/reference/catalog/pg_partition_rule.html.md.erb
new file mode 100644
index 0000000..9648132
--- /dev/null
+++ b/reference/catalog/pg_partition_rule.html.md.erb
@@ -0,0 +1,28 @@
+---
+title: pg_partition_rule
+---
+
+The `pg_partition_rule` system catalog table is used to track partitioned tables, their check constraints, and data containment rules. Each row of `pg_partition_rule` represents either a leaf partition (the bottom level partitions that contain data), or a branch partition (a top or mid-level partition that is used to define the partition hierarchy, but does not contain any data).
+
+<a id="topic1__hc179425"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_partition\_rule</span>
+
+
+| column | type | references | description |
+|---------------------|----------|----------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `paroid` | oid | pg\_partition.oid | Row identifier of the partitioning level (from [pg\_partition](pg_partition.html#topic1)) to which this partition belongs. In the case of a branch partition, the corresponding table (identified by `pg_partition_rule`) is an empty container table. In case of a leaf partition, the table contains the rows for that partition containment rule. |
+| `parchildrelid` | oid | pg\_class.oid | The table identifier of the partition (child table). |
+| `parparentrule` | oid | pg\_partition\_rule.paroid | The row identifier of the rule associated with the parent table of this partition. |
+| `parname` | name | | The given name of this partition. |
+| `parisdefault` | boolean | | Whether or not this partition is a default partition. |
+| `parruleord` | smallint | | For range partitioned tables, the rank of this partition on this level of the partition hierarchy. |
+| `parrangestartincl` | boolean | | For range partitioned tables, whether or not the starting value is inclusive. |
+| `parrangeendincl` | boolean | | For range partitioned tables, whether or not the ending value is inclusive. |
+| `parrangestart` | text | | For range partitioned tables, the starting value of the range. |
+| `parrangeend` | text | | For range partitioned tables, the ending value of the range. |
+| `parrangeevery` | text | | For range partitioned tables, the interval value of the `EVERY` clause. |
+| `parlistvalues` | text | | For list partitioned tables, the list of values assigned to this partition. |
+| `parreloptions` | text | | An array describing the storage characteristics of the particular partition. |
+
+
+
diff --git a/reference/catalog/pg_partition_templates.html.md.erb b/reference/catalog/pg_partition_templates.html.md.erb
new file mode 100644
index 0000000..ff397fb
--- /dev/null
+++ b/reference/catalog/pg_partition_templates.html.md.erb
@@ -0,0 +1,30 @@
+---
+title: pg_partition_templates
+---
+
+The `pg_partition_templates` system view is used to show the subpartitions that were created using a subpartition template.
+
+<a id="topic1__hd179967"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_partition\_templates</span>
+
+
+| column | type | references | description |
+|---------------------------|----------|------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `schemaname` | name | | The name of the schema the partitioned table is in. |
+| `tablename` | name | | The table name of the top-level parent table. |
+| `partitionname` | name | | The name of the subpartition (this is the name to use if referring to the partition in an `ALTER TABLE` command). `NULL` if the partition was not given a name at create time or generated by an `EVERY` clause. |
+| `partitiontype` | text | | The type of subpartition (range or list). |
+| `partitionlevel` | smallint | | The level of this subpartition in the hierarchy. |
+| `partitionrank` | bigint | | For range partitions, the rank of the partition compared to other partitions of the same level. |
+| `partitionposition` | smallint | | The rule order position of this subpartition. |
+| `partitionlistvalues` | text | | For list partitions, the list value(s) associated with this subpartition. |
+| `partitionrangestart` | text | | For range partitions, the start value of this subpartition. |
+| `partitionstartinclusive` | boolean | | `T` if the start value is included in this subpartition. `F` if it is excluded. |
+| `partitionrangeend` | text | | For range partitions, the end value of this subpartition. |
+| `partitionendinclusive` | boolean | | `T` if the end value is included in this subpartition. `F` if it is excluded. |
+| `partitioneveryclause` | text | | The `EVERY` clause (interval) of this subpartition. |
+| `partitionisdefault` | boolean | | `T` if this is a default subpartition, otherwise `F`. |
+| `partitionboundary` | text | | The entire partition specification for this subpartition. |
+
+
+
diff --git a/reference/catalog/pg_partitions.html.md.erb b/reference/catalog/pg_partitions.html.md.erb
new file mode 100644
index 0000000..2c0b26a
--- /dev/null
+++ b/reference/catalog/pg_partitions.html.md.erb
@@ -0,0 +1,30 @@
+---
+title: pg_partitions
+---
+
+The `pg_partitions` system view is used to show the structure of a partitioned table.
+
+<a id="topic1__he143898"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_partitions</span>
+
+| column | type | references | description |
+|----------------------------|----------|------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `schemaname` | name | | The name of the schema the partitioned table is in. |
+| `tablename` | name | | The name of the top-level parent table. |
+| `partitiontablename` | name | | The relation name of the partitioned table (this is the table name to use if accessing the partition directly). |
+| `partitionname` | name | | The name of the partition (this is the name to use if referring to the partition in an `ALTER TABLE` command). `NULL` if the partition was not given a name at create time or generated by an `EVERY` clause. |
+| `parentpartitiontablename` | name | | The relation name of the parent table one level up from this partition. |
+| `parentpartitionname` | name | | The given name of the parent table one level up from this partition. |
+| `partitiontype` | text | | The type of partition (range or list). |
+| `partitionlevel` | smallint | | The level of this partition in the hierarchy. |
+| `partitionrank` | bigint | | For range partitions, the rank of the partition compared to other partitions of the same level. |
+| `partitionposition` | smallint | | The rule order position of this partition. |
+| `partitionlistvalues` | text | | For list partitions, the list value(s) associated with this partition. |
+| `partitionrangestart` | text | | For range partitions, the start value of this partition. |
+| `partitionstartinclusive` | boolean | | `T` if the start value is included in this partition. `F` if it is excluded. |
+| `partitionrangeend` | text | | For range partitions, the end value of this partition. |
+| `partitionendinclusive` | boolean | | `T` if the end value is included in this partition. `F` if it is excluded. |
+| `partitioneveryclause` | text | | The `EVERY` clause (interval) of this partition. |
+| `partitionisdefault` | boolean | | `T` if this is a default partition, otherwise `F`. |
+| `partitionboundary` | text | | The entire partition specification for this partition. |
+
diff --git a/reference/catalog/pg_pltemplate.html.md.erb b/reference/catalog/pg_pltemplate.html.md.erb
new file mode 100644
index 0000000..4dda8ff
--- /dev/null
+++ b/reference/catalog/pg_pltemplate.html.md.erb
@@ -0,0 +1,22 @@
+---
+title: pg_pltemplate
+---
+
+The `pg_pltemplate` system catalog table stores template information for procedural languages. A template for a language allows the language to be created in a particular database by a simple `CREATE LANGUAGE` command, with no need to specify implementation details. Unlike most system catalogs, `pg_pltemplate` is shared across all databases of HAWQ system: there is only one copy of `pg_pltemplate` per system, not one per database. This allows the information to be accessible in each database as it is needed.
+
+There are not currently any commands that manipulate procedural language templates; to change the built-in information, a superuser must modify the table using ordinary `INSERT`, `DELETE`, or `UPDATE` commands.
+
+<a id="topic1__hf150092"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_pltemplate</span>
+
+| column | type | references | description |
+|------------------|-------------|------------|-------------------------------------------------------|
+| `tmplname` | name | | Name of the language this template is for |
+| `tmpltrusted` | boolean | | True if language is considered trusted |
+| `tmplhandler` | text | | Name of call handler function |
+| `tmplvalidator ` | text | | Name of validator function, or NULL if none |
+| `tmpllibrary` | text | | Path of shared library that implements language |
+| `tmplacl` | aclitem\[\] | | Access privileges for template (not yet implemented). |
+
+
+
diff --git a/reference/catalog/pg_proc.html.md.erb b/reference/catalog/pg_proc.html.md.erb
new file mode 100644
index 0000000..4d1d194
--- /dev/null
+++ b/reference/catalog/pg_proc.html.md.erb
@@ -0,0 +1,36 @@
+---
+title: pg_proc
+---
+
+The `pg_proc` system catalog table stores information about functions (or procedures), both built-in functions and those defined by `CREATE FUNCTION`. The table contains data for aggregate and window functions as well as plain functions. If `proisagg` is true, there should be a matching row in `pg_aggregate`. If `proiswin` is true, there should be a matching row in `pg_window`.
+
+For compiled functions, both built-in and dynamically loaded, `prosrc` contains the function's C-language name (link symbol). For all other currently-known language types, `prosrc` contains the function's source text. `probin` is unused except for dynamically-loaded C functions, for which it gives the name of the shared library file containing the function.
+
+<a id="topic1__hg150092"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_proc</span>
+
+| column | type | references | description |
+|------------------|-------------|-------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `proname` | name | | Name of the function. |
+| `pronamespace` | oid | pg\_namespace.oid | The OID of the namespace that contains this function. |
+| `proowner` | oid | pg\_authid.oid | Owner of the function. |
+| `prolang` | oid | pg\_language.oid | Implementation language or call interface of this function. |
+| `proisagg` | boolean | | Function is an aggregate function. |
+| `prosecdef` | boolean | | Function is a security definer (for example, a 'setuid' function). |
+| `proisstrict` | boolean | | Function returns NULL if any call argument is NULL. In that case the function will not actually be called at all. Functions that are not strict must be prepared to handle NULL inputs. |
+| `proretset` | boolean | | Function returns a set (multiple values of the specified data type). |
+| `provolatile` | char | | Tells whether the function's result depends only on its input arguments, or is affected by outside factors. `i` = *immutable* (always delivers the same result for the same inputs), `s` = *stable* (results (for fixed inputs) do not change within a scan), or `v` = *volatile* (results may change at any time or functions with side-effects). |
+| `pronargs` | smallint | | Number of arguments. |
+| `prorettype` | oid | pg\_type.oid | Data type of the return value. |
+| `proiswin` | boolean | | Function is neither an aggregate nor a scalar function, but a pure window function. |
+| `proargtypes` | oidvector | pg\_type.oid | An array with the data types of the function arguments. This includes only input arguments (including `INOUT` arguments), and thus represents the call signature of the function. |
+| `proallargtypes` | oid\[\] | pg\_type.oid | An array with the data types of the function arguments. This includes all arguments (including `OUT` and `INOUT` arguments); however, if all the arguments are `IN` arguments, this field will be null. Note that subscripting is 1-based, whereas for historical reasons proargtypes is subscripted from 0. |
+| `proargmodes` | char\[\] | | An array with the modes of the function arguments: `i` = `IN`, `o` = `OUT` , `b` = `INOUT`. If all the arguments are IN arguments, this field will be null. Note that subscripts correspond to positions of proallargtypes not proargtypes. |
+| `proargnames` | text\[\] | | An array with the names of the function arguments. Arguments without a name are set to empty strings in the array. If none of the arguments have a name, this field will be null. Note that subscripts correspond to positions of proallargtypes not proargtypes. |
+| `prosrc ` | text | | This tells the function handler how to invoke the function. It might be the actual source code of the function for interpreted languages, a link symbol, a file name, or just about anything else, depending on the implementation language/call convention. |
+| `probin` | bytea | | Additional information about how to invoke the function. Again, the interpretation is language-specific. |
+| `proacl` | aclitem\[\] | | Access privileges for the function as given by `GRANT`/`REVOKE`. |
+
+
+
+
diff --git a/reference/catalog/pg_resqueue.html.md.erb b/reference/catalog/pg_resqueue.html.md.erb
new file mode 100644
index 0000000..0b8d414
--- /dev/null
+++ b/reference/catalog/pg_resqueue.html.md.erb
@@ -0,0 +1,30 @@
+---
+title: pg_resqueue
+---
+
+The `pg_resqueue` system catalog table contains information about HAWQ resource queues, which are used for managing resources. This table is populated only on the master. This table is defined in the `pg_global` tablespace, meaning it is globally shared across all databases in the system.
+
+<a id="topic1__hi141982"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_resqueue</span>
+
+| column | type | references | description |
+|-------------------------|--------------------------|------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `rsqname` | name | | The name of the resource queue. |
+| `parentoid` | oid | | OID of the parent queue of the resource queue. |
+| `activestats` | integer | | The maximum number of parallel active statements allowed for the resource queue. |
+| `memorylimit` | text | | The maximum amount of memory that can be consumed by the resource queue (expressed as a percentage of the cluster's memory.) |
+| `corelimit` | text | | The maximum amount of cores that can be consumed by the resource queue (expressed as a percentage of the cluster's cores.) |
+| `resovercommit` | real | | The ratio of resource consumption overcommit for the resource queue. |
+| `allocpolicy` | text | | The resource allocation policy name for the resource queue. |
+| `vsegresourcequota` | text | | The virtual segment resource quota for the resource queue. |
+| `nvsegupperlimit` | integer | | The upper limit of number of virtual segments allowed for one statement execution. |
+| `nvseglowerlimit` | integer | | The lower limit of number of virtual segments allowed for one statement execution. |
+| `nvsegupperlimitperseg` | real | | The upper limit of number of virtual segments allowed for one statement execution. The limit is averaged by the number of segments in the cluster. |
+| `nvseglowerlimitperseg` | real | | The lower limit of number of virtual segments aloowed for one statement execution. The limit is averaged by the number of segments in the cluster. |
+| `creationtime` | timestamp with time zone | | Time when the resource queue was created. |
+| `updatetime` | timestamp with time zone | | Time when the resource queue was last changed. |
+| `status` | text | | Current status of the resource queue.Possible values are `branch`, which indicates a branch resource queue (has children), and `NULL`, which indicates a leaf-level queue (no children). |
+
+
+
+
diff --git a/reference/catalog/pg_resqueue_status.html.md.erb b/reference/catalog/pg_resqueue_status.html.md.erb
new file mode 100644
index 0000000..108fa36
--- /dev/null
+++ b/reference/catalog/pg_resqueue_status.html.md.erb
@@ -0,0 +1,94 @@
+---
+title: pg_resqueue_status
+---
+
+The `pg_resqueue_status` view allows administrators to see status and activity for a workload management resource queue. It shows how many queries are waiting to run and how many queries are currently active in the system from a particular resource queue.
+
+<a id="topic1__fp141982"></a>
+<span class="tablecap">Table 1. pg\_resqueue\_status</span>
+
+<table>
+<colgroup>
+<col width="25%" />
+<col width="25%" />
+<col width="25%" />
+<col width="25%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>column</th>
+<th>type</th>
+<th>references</th>
+<th>description</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td><code class="ph codeph">rsqname</code></td>
+<td>name</td>
+<td>pg_resqueue_ rsqname</td>
+<td>The name of the resource queue.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">segmem</code></td>
+<td>text</td>
+<td> </td>
+<td>The calculated virtual segment memory resource quota.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">segcore</code></td>
+<td>text</td>
+<td> </td>
+<td>The calculated virtual segment core resource quota.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">segsize</code></td>
+<td>text</td>
+<td> </td>
+<td>The number of virtual segments that can be allocated to the resource queue.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">segsizemax</code></td>
+<td>text</td>
+<td> </td>
+<td>The maximum number of virtual segments that can be allocated to the resource queue.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">inusemem</code></td>
+<td>text</td>
+<td> </td>
+<td>Aggregated in-use memory by running statements.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">inusecore</code></td>
+<td>text</td>
+<td> </td>
+<td>Aggregated in-use core by running statements.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">rsqholders</code></td>
+<td>text</td>
+<td> </td>
+<td>The number of resource holders for running statements. A resource holder is a running statement whose allocated resources from the resource manager has not been returned yet., In other words, the statement holds some resources allocated from the resource manager.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">resqwaiters</code></td>
+<td>text</td>
+<td> </td>
+<td>The number of resource requests that are queued and waiting for the resource.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">paused</code></td>
+<td>text</td>
+<td> </td>
+<td>The dynamic pause status of the resource queue. There are three possible statuses:
+<ul>
+<li><code class="ph codeph">T</code> : Queue is paused for the allocation of resources to queued and incoming requests.</li>
+<li><code class="ph codeph">F</code> : Queue is in a normal working status.</li>
+<li><code class="ph codeph">R</code> : Queue is paused and may have encountered resource fragmentation.</li>
+</ul></td>
+</tr>
+</tbody>
+</table>
+
+
diff --git a/reference/catalog/pg_rewrite.html.md.erb b/reference/catalog/pg_rewrite.html.md.erb
new file mode 100644
index 0000000..c42e855
--- /dev/null
+++ b/reference/catalog/pg_rewrite.html.md.erb
@@ -0,0 +1,20 @@
+---
+title: pg_rewrite
+---
+
+The `pg_rewrite` system catalog table stores rewrite rules for tables and views. `pg_class.relhasrules` must be true if a table has any rules in this catalog.
+
+<a id="topic1__hm149830"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_rewrite</span>
+
+| column | type | references | description |
+|--------------|----------|---------------|---------------------------------------------------------------------------------------------------------|
+| `rulename` | name | | Rule name. |
+| `ev_class` | oid | pg\_class.oid | The table this rule is for. |
+| `ev_attr` | smallint | | The column this rule is for (currently, always zero to indicate the whole table). |
+| `ev_type ` | char | | Event type that the rule is for: 1 = SELECT, 2 = UPDATE, 3 = INSERT, 4 = DELETE. |
+| `is_instead` | boolean | | True if the rule is an INSTEAD rule. |
+| `ev_qual` | text | | Expression tree (in the form of a `nodeToString()` representation) for the rule's qualifying condition. |
+| `ev_action` | text | | Query tree (in the form of a `nodeToString()` representation) for the rule's action. |
+
+
diff --git a/reference/catalog/pg_roles.html.md.erb b/reference/catalog/pg_roles.html.md.erb
new file mode 100644
index 0000000..9e70f46
--- /dev/null
+++ b/reference/catalog/pg_roles.html.md.erb
@@ -0,0 +1,31 @@
+---
+title: pg_roles
+---
+
+The view `pg_roles` provides access to information about database roles. This is simply a publicly readable view of [pg\_authid](pg_authid.html#topic1) that blanks out the password field. This view explicitly exposes the OID column of the underlying table, since that is needed to do joins to other catalogs.
+
+<a id="topic1__hn141982"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_roles</span>
+
+| column | type | references | description |
+|---------------------|--------------------------|------------------|---------------------------------------------------------------------------------------------------------------------|
+| `rolname` | name | | Role name |
+| `rolsuper` | boolean | | Role has superuser privileges |
+| `rolinherit` | boolean | | Role automatically inherits privileges of roles it is a member of |
+| `rolcreaterole` | boolean | | Role may create more roles |
+| `rolcreatedb` | boolean | | Role may create databases |
+| `rolcatupdate` | boolean | | Role may update system catalogs directly. (Even a superuser may not do this unless this column is true.) |
+| `rolcanlogin` | boolean | | Role may log in. That is, this role can be given as the initial session authorization identifier |
+| `rolconnlimit` | integer | | For roles that can log in, this sets maximum number of concurrent connections this role can make. -1 means no limit |
+| `rolpassword` | text | | Not the password (always reads as \*\*\*\*\*\*\*\*) |
+| `rolvaliduntil ` | timestamp with time zone | | Password expiry time (only used for password authentication); NULL if no expiration |
+| `rolconfig ` | text\[\] | | Session defaults for run-time configuration variables |
+| ` rolresqueue` | oid | pg\_resqueue.oid | Object ID of the resource queue this role is assigned to. |
+| `oid` | oid | pg\_authid.oid | Object ID of role |
+| `rolcreaterextgpfd` | boolean | | Role may create readable external tables that use the gpfdist protocol. |
+| `rolcreaterexthttp` | boolean | | Role may create readable external tables that use the http protocol. |
+| `rolcreatewextgpfd` | boolean | | Role may create writable external tables that use the gpfdist protocol. |
+
+
+
+
diff --git a/reference/catalog/pg_shdepend.html.md.erb b/reference/catalog/pg_shdepend.html.md.erb
new file mode 100644
index 0000000..b966155
--- /dev/null
+++ b/reference/catalog/pg_shdepend.html.md.erb
@@ -0,0 +1,28 @@
+---
+title: pg_shdepend
+---
+
+The `pg_shdepend` system catalog table records the dependency relationships between database objects and shared objects, such as roles. This information allows HAWQ to ensure that those objects are unreferenced before attempting to delete them. See also [pg\_depend](pg_depend.html#topic1), which performs a similar function for dependencies involving objects within a single database. Unlike most system catalogs, `pg_shdepend` is shared across all databases of HAWQ system: there is only one copy of `pg_shdepend` per system, not one per database.
+
+In all cases, a `pg_shdepend` entry indicates that the referenced object may not be dropped without also dropping the dependent object. However, there are several subflavors identified by `deptype`:
+
+- **SHARED\_DEPENDENCY\_OWNER (o)** — The referenced object (which must be a role) is the owner of the dependent object.
+- **SHARED\_DEPENDENCY\_ACL (a)** — The referenced object (which must be a role) is mentioned in the ACL (access control list) of the dependent object.
+- **SHARED\_DEPENDENCY\_PIN (p)** — There is no dependent object; this type of entry is a signal that the system itself depends on the referenced object, and so that object must never be deleted. Entries of this type are created only by system initialization. The columns for the dependent object contain zeroes. <a id="topic1__ho143898"></a>
+
+<span class="tablecap">Table 1. pg\_catalog.pg\_shdepend</span>
+
+| column | type | references | description |
+|----------------|---------|------------------|------------------------------------------------------------------------------------------------------------|
+| `dbid` | oid | pg\_database.oid | The OID of the database the dependent object is in, or zero for a shared object. |
+| `classid` | oid | pg\_class.oid | The OID of the system catalog the dependent object is in. |
+| `objid` | oid | any OID column | The OID of the specific dependent object. |
+| `objsubid ` | integer | | For a table column, this is the column number. For all other object types, this column is zero. |
+| `refclassid` | oid | pg\_class.oid | The OID of the system catalog the referenced object is in (must be a shared catalog). |
+| `refobjid` | oid | any OID column | The OID of the specific referenced object. |
+| `refobjsubid ` | integer | | For a table column, this is the referenced column number. For all other object types, this column is zero. |
+| `deptype` | char | | A code defining the specific semantics of this dependency relationship. |
+
+
+
+
diff --git a/reference/catalog/pg_shdescription.html.md.erb b/reference/catalog/pg_shdescription.html.md.erb
new file mode 100644
index 0000000..133e326
--- /dev/null
+++ b/reference/catalog/pg_shdescription.html.md.erb
@@ -0,0 +1,18 @@
+---
+title: pg_shdescription
+---
+
+The `pg_shdescription` system catalog table stores optional descriptions (comments) for shared database objects. Descriptions can be manipulated with the `COMMENT` command and viewed with `psql`'s `\d` meta-commands. See also [pg\_description](pg_description.html#topic1), which performs a similar function for descriptions involving objects within a single database. Unlike most system catalogs, `pg_shdescription` is shared across all databases of a HAWQ system: there is only one copy of `pg_shdescription` per system, not one per database.
+
+<a id="topic1__hp143898"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_shdescription</span>
+
+
+| column | type | references | description |
+|---------------|------|----------------|---------------------------------------------------------------|
+| `objoid` | oid | any OID column | The OID of the object this description pertains to. |
+| `classoid` | oid | pg\_class.oid | The OID of the system catalog this object appears in |
+| `description` | text | | Arbitrary text that serves as the description of this object. |
+
+
+
diff --git a/reference/catalog/pg_stat_activity.html.md.erb b/reference/catalog/pg_stat_activity.html.md.erb
new file mode 100644
index 0000000..008ae8b
--- /dev/null
+++ b/reference/catalog/pg_stat_activity.html.md.erb
@@ -0,0 +1,30 @@
+---
+title: pg_stat_activity
+---
+
+The view `pg_stat_activity` shows one row per server process and details about it associated user session and query. The columns that report data on the current query are available unless the parameter `stats_command_string` has been turned off. Furthermore, these columns are only visible if the user examining the view is a superuser or the same as the user owning the process being reported on.
+
+<a id="topic1__hq141982"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_stat\_activity</span>
+
+| column | type | references | description |
+|--------------------|--------------------------|------------------|---------------------------------------------------------------|
+| `datid` | oid | pg\_database.oid | Database OID |
+| `datname` | name | | Database name |
+| `procpid` | integer | | Process ID of the server process |
+| `sess_id` | integer | | Session ID |
+| `usesysid` | oid | pg\_authid.oid | Role OID |
+| `usename` | name | | Role name |
+| `current_query` | text | | Current query that process is running |
+| `waiting` | boolean | | True if waiting on a lock, false if not waiting |
+| `query_start` | timestamp with time zone | | Time query began execution |
+| `backend_start` | timestamp with time zone | | Time backend process was started |
+| `client_addr` | inet | | Client address |
+| `client_port` | integer | | Client port |
+| `application_name` | text | | Client application name |
+| `xact_start` | timestamp with time zone | | Transaction start time |
+| `waiting_resource` | boolean | | True if waiting for resource allocation, false if not waiting |
+
+
+
+
diff --git a/reference/catalog/pg_stat_last_operation.html.md.erb b/reference/catalog/pg_stat_last_operation.html.md.erb
new file mode 100644
index 0000000..2f842a1
--- /dev/null
+++ b/reference/catalog/pg_stat_last_operation.html.md.erb
@@ -0,0 +1,21 @@
+---
+title: pg_stat_last_operation
+---
+
+The pg\_stat\_last\_operation table contains metadata tracking information about database objects (tables, views, etc.).
+
+<a id="topic1__hr138428"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_stat\_last\_operation</span>
+
+| column | type | references | description |
+|-----------------|-------------------------|----------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `classid` | oid | pg\_class.oid | OID of the system catalog containing the object. |
+| `objid` | oid | any OID column | OID of the object within its system catalog. |
+| `staactionname` | name | | The action that was taken on the object. |
+| `stasysid` | oid | pg\_authid.oid | A foreign key to pg\_authid.oid. |
+| `stausename` | name | | The name of the role that performed the operation on this object. |
+| `stasubtype` | text | | The type of object operated on or the subclass of operation performed. |
+| `statime` | timestamp with timezone | | The timestamp of the operation. This is the same timestamp that is written to the HAWQ server log files in case you need to look up more detailed information about the operation in the logs. |
+
+
+
diff --git a/reference/catalog/pg_stat_last_shoperation.html.md.erb b/reference/catalog/pg_stat_last_shoperation.html.md.erb
new file mode 100644
index 0000000..1e6785d
--- /dev/null
+++ b/reference/catalog/pg_stat_last_shoperation.html.md.erb
@@ -0,0 +1,23 @@
+---
+title: pg_stat_last_shoperation
+---
+
+The pg\_stat\_last\_shoperation table contains metadata tracking information about global objects (roles, tablespaces, etc.).
+
+<a id="topic1__hs138428"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_stat\_last\_shoperation</span>
+
+
+| column | type | references | description |
+|-----------------|-------------------------|----------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| classid | oid | pg\_class.oid | OID of the system catalog containing the object. |
+| `objid` | oid | any OID column | OID of the object within its system catalog. |
+| `staactionname` | name | | The action that was taken on the object. |
+| `stasysid` | oid | | |
+| `stausename` | name | | The name of the role that performed the operation on this object. |
+| `stasubtype` | text | | The type of object operated on or the subclass of operation performed. |
+| `statime` | timestamp with timezone | | The timestamp of the operation. This is the same timestamp that is written to the HAWQ server log files in case you need to look up more detailed information about the operation in the logs. |
+
+
+
+
diff --git a/reference/catalog/pg_stat_operations.html.md.erb b/reference/catalog/pg_stat_operations.html.md.erb
new file mode 100644
index 0000000..d31c1e0
--- /dev/null
+++ b/reference/catalog/pg_stat_operations.html.md.erb
@@ -0,0 +1,87 @@
+---
+title: pg_stat_operations
+---
+
+The view `pg_stat_operations` shows details about the last operation performed on a database object (such as a table, index, view or database) or a global object (such as a role).
+
+<a id="topic1__ht141982"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_stat\_operations</span>
+
+
+<table>
+<colgroup>
+<col width="25%" />
+<col width="25%" />
+<col width="25%" />
+<col width="25%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>column</th>
+<th>type</th>
+<th>references</th>
+<th>description</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td><code class="ph codeph">classname</code></td>
+<td>text</td>
+<td> </td>
+<td>The name of the system table in the <code class="ph codeph">pg_catalog</code> schema where the record about this object is stored (<code class="ph codeph">pg_class</code>=relations, <code class="ph codeph">pg_database</code>=databases,
+<p><code class="ph codeph">pg_namespace</code>=schemas,</p>
+<p><code class="ph codeph">pg_authid</code>=roles)</p></td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">objname</code></td>
+<td>name</td>
+<td> </td>
+<td>The name of the object.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">objid</code></td>
+<td>oid</td>
+<td> </td>
+<td>The OID of the object.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">schemaname</code></td>
+<td>name</td>
+<td> </td>
+<td>The name of the schema where the object resides.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">usestatus</code></td>
+<td>text</td>
+<td> </td>
+<td>The status of the role who performed the last operation on the object (<code class="ph codeph">CURRENT</code>=a currently active role in the system, <code class="ph codeph">DROPPED</code>=a role that no longer exists in the system, <code class="ph codeph">CHANGED</code>=a role name that exists in the system, but has changed since the last operation was performed).</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">usename</code></td>
+<td>name</td>
+<td> </td>
+<td>The name of the role that performed the operation on this object.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">actionname</code></td>
+<td>name</td>
+<td> </td>
+<td>The action that was taken on the object.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">subtype</code></td>
+<td>text</td>
+<td> </td>
+<td>The type of object operated on or the subclass of operation performed.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">statime</code></td>
+<td>timestamp with time zone</td>
+<td> </td>
+<td>The timestamp of the operation. This is the same timestamp that is written to the HAWQ server log files in case you need to look up more detailed information about the operation in the logs.</td>
+</tr>
+</tbody>
+</table>
+
+
+
diff --git a/reference/catalog/pg_stat_partition_operations.html.md.erb b/reference/catalog/pg_stat_partition_operations.html.md.erb
new file mode 100644
index 0000000..2d2fb17
--- /dev/null
+++ b/reference/catalog/pg_stat_partition_operations.html.md.erb
@@ -0,0 +1,28 @@
+---
+title: pg_stat_partition_operations
+---
+
+The `pg_stat_partition_operations` view shows details about the last operation performed on a partitioned table.
+
+<a id="topic1__hu141982"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_stat\_partition\_operations</span>
+
+| column | type | references | description |
+|--------------------|--------------------------|------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `classname` | text | | The name of the system table in the `pg_catalog` schema where the record about this object is stored (always `pg_class` for tables and partitions). |
+| `objname` | name | | The name of the object. |
+| `objid` | oid | | The OID of the object. |
+| `schemaname` | name | | The name of the schema where the object resides. |
+| `usestatus` | text | | The status of the role who performed the last operation on the object (`CURRENT`=a currently active role in the system, `DROPPED`=a role that no longer exists in the system, `CHANGED`=a role name that exists in the system, but its definition has changed since the last operation was performed). |
+| `usename` | name | | The name of the role that performed the operation on this object. |
+| `actionname` | name | | The action that was taken on the object. |
+| `subtype` | text | | The type of object operated on or the subclass of operation performed. |
+| `statime` | timestamp with time zone | | The timestamp of the operation. This is the same timestamp that is written to the HAWQ server log files in case you need to look up more detailed information about the operation in the logs. |
+| `partitionlevel` | smallint | | The level of this partition in the hierarchy. |
+| `parenttablename` | name | | The relation name of the parent table one level up from this partition. |
+| `parentschemaname` | name | | The name of the schema where the parent table resides. |
+| `parent_relid` | oid | | The OID of the parent table one level up from this partition. |
+
+
+
+
diff --git a/reference/catalog/pg_statistic.html.md.erb b/reference/catalog/pg_statistic.html.md.erb
new file mode 100644
index 0000000..b784da1
--- /dev/null
+++ b/reference/catalog/pg_statistic.html.md.erb
@@ -0,0 +1,30 @@
+---
+title: pg_statistic
+---
+
+The `pg_statistic` system catalog table stores statistical data about the contents of the database. Entries are created by `ANALYZE` and subsequently used by the query optimizer. There is one entry for each table column that has been analyzed. Note that all the statistical data is inherently approximate, even assuming that it is up-to-date.
+
+`pg_statistic` also stores statistical data about the values of index expressions. These are described as if they were actual data columns; in particular, `starelid` references the index. No entry is made for an ordinary non-expression index column, however, since it would be redundant with the entry for the underlying table column.
+
+Since different kinds of statistics may be appropriate for different kinds of data, `pg_statistic` is designed not to assume very much about what sort of statistics it stores. Only extremely general statistics (such as nullness) are given dedicated columns in `pg_statistic`. Everything else is stored in slots, which are groups of associated columns whose content is identified by a code number in one of the slot's columns.
+
+`pg_statistic` should not be readable by the public, since even statistical information about a table's contents may be considered sensitive (for example: minimum and maximum values of a salary column). `pg_stats` is a publicly readable view on `pg_statistic` that only exposes information about those tables that are readable by the current user. See [pg\_stats](pg_stats.html#topic1), for more information on this view.
+
+<a id="topic1__hv156260"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_statistic</span>
+
+| column | type | references | description |
+|---------------|----------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `starelid` | oid | pg\_class.oid | The table or index that the described column belongs to. |
+| `staattnum` | smallint | pg\_attribute.attnum | The number of the described column. |
+| `stanullfrac` | real | | The fraction of the column's entries that are null. |
+| `stawidth` | integer | | The average stored width, in bytes, of nonnull entries. |
+| `stadistinct` | real | | The number of distinct nonnull data values in the column. A value greater than zero is the actual number of distinct values. A value less than zero is the negative of a fraction of the number of rows in the table (for example, a column in which values appear about twice on the average could be represented by `stadistinct` = -0.5). A zero value means the number of distinct values is unknown. |
+| `stakindN` | smallint | | A code number indicating the kind of statistics stored in the `N`th slot of the `pg_statistic` row. |
+| `staopN` | oid | pg\_operator.oid | An operator used to derive the statistics stored in the `N`th slot. For example, a histogram slot would show the `<` operator that defines the sort order of the data. |
+| `stanumbersN` | real\[\] | | Numerical statistics of the appropriate kind for the `N`th slot, or NULL if the slot kind does not involve numerical values. |
+| `stavaluesN` | anyarray | | Column data values of the appropriate kind for the `N`th slot, or NULL if the slot kind does not store any data values. Each array's element values are actually of the specific column's data type, so there is no way to define these columns' type more specifically than `anyarray`. |
+
+
+
+
diff --git a/reference/catalog/pg_stats.html.md.erb b/reference/catalog/pg_stats.html.md.erb
new file mode 100644
index 0000000..a14c4d2
--- /dev/null
+++ b/reference/catalog/pg_stats.html.md.erb
@@ -0,0 +1,27 @@
+---
+title: pg_stats
+---
+
+The `pg_stats` is a publicly readable view on `pg_statistic` that only exposes information about those tables that are readable by the current user. All the statistical data is inherently approximate, even assuming that it is up-to-date.The `pg_stats` view presents the contents of `pg_statistic` in a friendlier format.
+
+All the statistical data is inherently approximate, even assuming that it is up-to-date.The `pg_stats` schema must be extended whenever new slot types are defined.
+
+<a id="topic1__table_ckx_t2w_jv"></a>
+<span class="tablecap">Table 1. pg\_stats</span>
+
+| Name | Type | References | Description |
+|---------------------|----------|----------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| schemaname | name | [pg\_namespace](pg_namespace.html#topic1).nspname. | The name of the schema containing the table. |
+| tablename | name | [pg\_class](pg_class.html#topic1).relname | The name of the table. |
+| attname | name | [pg\_attribute](pg_attribute.html#topic1).attname | The name of the column this row describes. |
+| null\_frac | real | | The fraction of column entries that are null. |
+| avg\_width | integer | | The average storage width in bytes of the column's entries, calculated as `avg(pg_column_size(column_name))`. |
+| n\_distinct | real | | A positive number is an estimate of the number of distinct values in the column; the number is not expected to vary with the number of rows. A negative value is the number of distinct values divided by the number of rows, that is, the ratio of rows with distinct values for the column, negated. This form is used when the number of distinct values increases with the number of rows. A unique column, for example, has an `n_distinct` value of -1.0. Columns with an average width greater than 1024 are considered unique. |
+| most\_common-vals | anyarray | | An array containing the most common values in the column, or null if no values seem to be more common. If the `n_distinct` column is -1, `most_common_vals` is null. The length of the array is the lesser of the number of actual distinct column values or the value of the `default_statistics_target` configuration parameter. The number of values can be overridden for a column using `ALTER TABLE table SET COLUMN column SET STATISTICS N`. |
+| most\_common\_freqs | real\[\] | | An array containing the frequencies of the values in the `most_common_vals` array. This is the number of occurrences of the value divided by the total number of rows. The array is the same length as the `most_common_vals` array. It is null if `most_common_vals` is null. |
+| histogram\_bounds | anyarray | | An array of values that divide the column values into groups of approximately the same size. A histogram can be defined only if there is a `max()` aggregate function for the column. The number of groups in the histogram is the same as the `most_common_vals` array size. |
+| correlation | real | | HAWQ does not calculate the correlation statistic. |
+
+
+
+
diff --git a/reference/catalog/pg_tablespace.html.md.erb b/reference/catalog/pg_tablespace.html.md.erb
new file mode 100644
index 0000000..493c6b0
--- /dev/null
+++ b/reference/catalog/pg_tablespace.html.md.erb
@@ -0,0 +1,22 @@
+---
+title: pg_tablespace
+---
+
+The `pg_tablespace` system catalog table stores information about the available tablespaces. Tables can be placed in particular tablespaces to aid administration of disk layout. Unlike most system catalogs, `pg_tablespace` is shared across all databases of a HAWQ system: there is only one copy of `pg_tablespace` per system, not one per database.
+
+<a id="topic1__hx156260"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_tablespace</span>
+
+| column | type | references | description |
+|-------------------|-------------|-------------------|-----------------------------------------------------------------------------------------------------------------------------|
+| `spcname` | name | | Tablespace name. |
+| `spcowner` | oid | pg\_authid.oid | Owner of the tablespace, usually the user who created it. |
+| `spclocation` | text\[\] | | Deprecated. |
+| `spcacl ` | aclitem\[\] | | Tablespace access privileges. |
+| `spcprilocations` | text\[\] | | Deprecated. |
+| `spcmrilocations` | text\[\] | | Deprecated. |
+| `spcfsoid` | oid | pg\_filespace.oid | The object id of the filespace used by this tablespace. A filespace defines directory locations on the master and segments. |
+
+
+
+
diff --git a/reference/catalog/pg_trigger.html.md.erb b/reference/catalog/pg_trigger.html.md.erb
new file mode 100644
index 0000000..3074e46
--- /dev/null
+++ b/reference/catalog/pg_trigger.html.md.erb
@@ -0,0 +1,114 @@
+---
+title: pg_trigger
+---
+
+The `pg_trigger` system catalog table stores triggers on tables.
+
+**Note:** HAWQ does not support triggers.
+
+<a id="topic1__hy183441"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_trigger</span>
+
+<table>
+<colgroup>
+<col width="25%" />
+<col width="25%" />
+<col width="25%" />
+<col width="25%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>column</th>
+<th>type</th>
+<th>references</th>
+<th>description</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td><code class="ph codeph">tgrelid</code></td>
+<td>oid</td>
+<td><em>pg_class.oid</em>
+<p>Note that HAWQ does not enforce referential integrity.</p></td>
+<td>The table this trigger is on.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">tgname</code></td>
+<td>name</td>
+<td> </td>
+<td>Trigger name (must be unique among triggers of same table).</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">tgfoid</code></td>
+<td>oid</td>
+<td><em>pg_proc.oid</em>
+<p>Note that HAWQ does not enforce referential integrity.</p></td>
+<td>The function to be called.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">tgtype</code></td>
+<td>smallint</td>
+<td> </td>
+<td>Bit mask identifying trigger conditions.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">tgenabled</code></td>
+<td>boolean</td>
+<td> </td>
+<td>True if trigger is enabled.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">tgisconstraint</code></td>
+<td>boolean</td>
+<td> </td>
+<td>True if trigger implements a referential integrity constraint.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">tgconstrname</code></td>
+<td>name</td>
+<td> </td>
+<td>Referential integrity constraint name.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">tgconstrrelid</code></td>
+<td>oid</td>
+<td><em>pg_class.oid</em>
+<p>Note that HAWQ does not enforce referential integrity.</p></td>
+<td>The table referenced by an referential integrity constraint.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">tgdeferrable</code></td>
+<td>boolean</td>
+<td> </td>
+<td>True if deferrable.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">tginitdeferred</code></td>
+<td>boolean</td>
+<td> </td>
+<td>True if initially deferred.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">tgnargs</code></td>
+<td>smallint</td>
+<td> </td>
+<td>Number of argument strings passed to trigger function.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">tgattr</code></td>
+<td>int2vector</td>
+<td> </td>
+<td>Currently not used.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">tgargs</code></td>
+<td>bytea</td>
+<td> </td>
+<td>Argument strings to pass to trigger, each NULL-terminated.</td>
+</tr>
+</tbody>
+</table>
+
+
+
+
diff --git a/reference/catalog/pg_type.html.md.erb b/reference/catalog/pg_type.html.md.erb
new file mode 100644
index 0000000..e2ea28a
--- /dev/null
+++ b/reference/catalog/pg_type.html.md.erb
@@ -0,0 +1,176 @@
+---
+title: pg_type
+---
+
+The `pg_type` system catalog table stores information about data types. Base types (scalar types) are created with `CREATE TYPE`, and domains with `CREATE DOMAIN`. A composite type is automatically created for each table in the database, to represent the row structure of the table. It is also possible to create composite types with `CREATE TYPE AS`.
+
+<a id="topic1__hz156260"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_type</span>
+
+<table>
+<colgroup>
+<col width="25%" />
+<col width="25%" />
+<col width="25%" />
+<col width="25%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>column</th>
+<th>type</th>
+<th>references</th>
+<th>description</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td><code class="ph codeph">typname</code></td>
+<td>name</td>
+<td> </td>
+<td>Data type name.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">typnamespace</code></td>
+<td>oid</td>
+<td>pg_namespace.oid</td>
+<td>The OID of the namespace that contains this type.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">typowner</code></td>
+<td>oid</td>
+<td>pg_authid.oid</td>
+<td>Owner of the type.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">typlen</code></td>
+<td>smallint</td>
+<td> </td>
+<td>For a fixed-size type, <code class="ph codeph">typlen</code> is the number of bytes in the internal representation of the type. But for a variable-length type, <code class="ph codeph">typlen</code> is negative. <code class="ph codeph">-1</code> indicates a 'varlena' type (one that has a length word), <code class="ph codeph">-2</code> indicates a null-terminated C string.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">typbyval</code></td>
+<td>boolean</td>
+<td> </td>
+<td>Determines whether internal routines pass a value of this type by value or by reference. <code class="ph codeph">typbyval </code>had better be false if <code class="ph codeph">typlen</code> is not 1, 2, or 4 (or 8 on machines where Datum is 8 bytes). Variable-length types are always passed by reference. Note that <code class="ph codeph">typbyval</code> can be false even if the length would allow pass-by-value; this is currently true for type <code class="ph codeph">float4</code>, for example.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">typtype</code></td>
+<td>char</td>
+<td> </td>
+<td><code class="ph codeph">b</code> for a base type, <code class="ph codeph">c</code> for a composite type, <code class="ph codeph">d</code> for a domain, or <code class="ph codeph">p</code> for a pseudo-type.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">typisdefined</code></td>
+<td>boolean</td>
+<td> </td>
+<td>True if the type is defined, false if this is a placeholder entry for a not-yet-defined type. When false, nothing except the type name, namespace, and OID can be relied on.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">typdelim</code></td>
+<td>char</td>
+<td> </td>
+<td>Character that separates two values of this type when parsing array input. Note that the delimiter is associated with the array element data type, not the array data type.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">typrelid</code></td>
+<td>oid</td>
+<td>pg_class.oid</td>
+<td>If this is a composite type, then this column points to the <code class="ph codeph">pg_class</code> entry that defines the corresponding table. (For a free-standing composite type, the <code class="ph codeph">pg_class</code> entry does not really represent a table, but it is needed anyway for the type's <code class="ph codeph">pg_attribute</code> entries to link to.) Zero for non-composite types.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">typelem</code></td>
+<td>oid</td>
+<td>pg_type.oid</td>
+<td>If not <code class="ph codeph">0</code> then it identifies another row in pg_type. The current type can then be subscripted like an array yielding values of type <code class="ph codeph">typelem</code>. A true array type is variable length (<code class="ph codeph">typlen</code> = <code class="ph codeph">-1</code>), but some fixed-length (<code class="ph codeph">tylpen</code> > <code class="ph codeph">0</code>) types also have nonzero <code class="ph codeph">typelem</code>, for example <code class="ph codeph">name</code> and <code class="ph codeph">point</code>. If a fixed-length type has a <code class="ph codeph">typelem</code> then its internal representation must be some number of values of the <code class="ph codeph">typelem</code> data type with no other data. Variable-length array types have a header defined by the array subroutines.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">typinput</code></td>
+<td>regproc</td>
+<td>pg_proc.oid</td>
+<td>Input conversion function (text format).</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">typoutput</code></td>
+<td>regproc</td>
+<td>pg_proc.oid</td>
+<td>Output conversion function (text format).</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">typreceive</code></td>
+<td>regproc</td>
+<td>pg_proc.oid</td>
+<td>Input conversion function (binary format), or 0 if none.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">typsend</code></td>
+<td>regproc</td>
+<td>pg_proc.oid</td>
+<td>Output conversion function (binary format), or 0 if none.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">typanalyze</code></td>
+<td>regproc</td>
+<td>pg_proc.oid</td>
+<td>Custom <code class="ph codeph">ANALYZE</code> function, or 0 to use the standard function.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">typalign</code></td>
+<td>char</td>
+<td> </td>
+<td>The alignment required when storing a value of this type. It applies to storage on disk as well as most representations of the value inside HAWQ. When multiple values are stored consecutively, such as in the representation of a complete row on disk, padding is inserted before a datum of this type so that it begins on the specified boundary. The alignment reference is the beginning of the first datum in the sequence. Possible values are:
+<p><code class="ph codeph">c</code> = char alignment (no alignment needed).</p>
+<p><code class="ph codeph">s</code> = short alignment (2 bytes on most machines).</p>
+<p><code class="ph codeph">i</code> = int alignment (4 bytes on most machines).</p>
+<p><code class="ph codeph">d</code> = double alignment (8 bytes on many machines, but not all).</p></td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">typstorage</code></td>
+<td>char</td>
+<td> </td>
+<td>For varlena types (those with <code class="ph codeph">typlen</code> = -1) tells if the type is prepared for toasting and what the default strategy for attributes of this type should be. Possible values are:
+<p><code class="ph codeph">p</code>: Value must always be stored plain.</p>
+<p><code class="ph codeph">e</code>: Value can be stored in a secondary relation (if relation has one, see <code class="ph codeph">pg_class.reltoastrelid</code>).</p>
+<p><code class="ph codeph">m</code>: Value can be stored compressed inline.</p>
+<p><code class="ph codeph">x</code>: Value can be stored compressed inline or stored in secondary storage.</p>
+<p>Note that <code class="ph codeph">m</code> columns can also be moved out to secondary storage, but only as a last resort (<code class="ph codeph">e</code> and <code class="ph codeph">x</code> columns are moved first).</p></td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">typnotnull</code></td>
+<td>boolean</td>
+<td> </td>
+<td>Represents a not-null constraint on a type. Used for domains only.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">typbasetype</code></td>
+<td>oid</td>
+<td>pg_type.oid</td>
+<td>Identifies the type that a domain is based on. Zero if this type is not a domain.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">typtypmod</code></td>
+<td>integer</td>
+<td> </td>
+<td>Domains use typtypmod to record the typmod to be applied to their base type (-1 if base type does not use a typmod). -1 if this type is not a domain.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">typndims</code></td>
+<td>integer</td>
+<td> </td>
+<td>The number of array dimensions for a domain that is an array (if <code class="ph codeph">typbasetype</code> is an array type; the domain's <code class="ph codeph">typelem</code> will match the base type's <code class="ph codeph">typelem</code>). Zero for types other than array domains.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">typdefaultbin</code></td>
+<td>text</td>
+<td> </td>
+<td>If not null, it is the <code class="ph codeph">nodeToString()</code> representation of a default expression for the type. This is only used for domains.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">typdefault</code></td>
+<td>text</td>
+<td> </td>
+<td>Null if the type has no associated default value. If not null, typdefault must contain a human-readable version of the default expression represented by typdefaultbin. If typdefaultbin is null and typdefault is not, then typdefault is the external representation of the type's default value, which may be fed to the type's input converter to produce a constant.</td>
+</tr>
+</tbody>
+</table>
+
+
diff --git a/reference/catalog/pg_type_encoding.html.md.erb b/reference/catalog/pg_type_encoding.html.md.erb
new file mode 100644
index 0000000..e6c3e7a
--- /dev/null
+++ b/reference/catalog/pg_type_encoding.html.md.erb
@@ -0,0 +1,15 @@
+---
+title: pg_type_encoding
+---
+
+The `pg_type_encoding` system catalog table contains the column storage type information.
+
+<a id="topic1__ia177831"></a>
+span class="tablecap">Table 1. pg\_catalog.pg\_type\_encoding</span>
+
+| column | type | modifers | storage | description |
+|--------------|------------|----------|----------|----------------------------------------------------------------------------------|
+| `typeid` | oid | not null | plain | Foreign key to [pg\_attribute](pg_attribute.html#topic1) |
+| `typoptions` | text \[ \] | | extended | The actual options |
+
+
diff --git a/reference/catalog/pg_window.html.md.erb b/reference/catalog/pg_window.html.md.erb
new file mode 100644
index 0000000..afe4c0d
--- /dev/null
+++ b/reference/catalog/pg_window.html.md.erb
@@ -0,0 +1,97 @@
+---
+title: pg_window
+---
+
+The `pg_window` table stores information about window functions. Window functions are often used to compose complex OLAP (online analytical processing) queries. Window functions are applied to partitioned result sets within the scope of a single query expression. A window partition is a subset of rows returned by a query, as defined in a special `OVER()` clause. Typical window functions are `rank`, `dense_rank`, and `row_number`. Each entry in `pg_window` is an extension of an entry in [pg\_proc](pg_proc.html#topic1). The [pg\_proc](pg_proc.html#topic1) entry carries the window function's name, input and output data types, and other information that is similar to ordinary functions.
+
+<a id="topic1__ic143898"></a>
+<span class="tablecap">Table 1. pg\_catalog.pg\_window</span>
+
+<table>
+<colgroup>
+<col width="25%" />
+<col width="25%" />
+<col width="25%" />
+<col width="25%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>column</th>
+<th>type</th>
+<th>references</th>
+<th>description</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td><code class="ph codeph">winfnoid</code></td>
+<td>regproc</td>
+<td>pg_proc.oid</td>
+<td>The OID in <code class="ph codeph">pg_proc</code> of the window function.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">winrequireorder</code></td>
+<td>boolean</td>
+<td> </td>
+<td>The window function requires its window specification to have an <code class="ph codeph">ORDER BY</code> clause.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">winallowframe</code></td>
+<td>boolean</td>
+<td> </td>
+<td>The window function permits its window specification to have a <code class="ph codeph">ROWS</code> or <code class="ph codeph">RANGE</code> framing clause.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">winpeercount</code></td>
+<td>boolean</td>
+<td> </td>
+<td>The peer group row count is required to compute this window function, so the Window node implementation must 'look ahead' as necessary to make this available in its internal state.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">wincount</code></td>
+<td>boolean</td>
+<td> </td>
+<td>The partition row count is required to compute this window function.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">winfunc</code></td>
+<td>regproc</td>
+<td>pg_proc.oid</td>
+<td>The OID in <code class="ph codeph">pg_proc</code> of a function to compute the value of an immediate-type window function.</td>
+</tr>
+<tr class="odd">
+<td><code class="ph codeph">winprefunc</code></td>
+<td>regproc</td>
+<td>pg_proc.oid</td>
+<td>The OID in <code class="ph codeph">pg_proc</code> of a preliminary window function to compute the partial value of a deferred-type window function.</td>
+</tr>
+<tr class="even">
+<td><code class="ph codeph">winpretype</code></td>
+<td>oid</td>
+<td>pg_type.oid</td>
+<td>The OID in <code class="ph codeph">pg_type</code> of the preliminary window function's result type.</td>
+</tr>
+<tr class="odd">
+<td>winfinfunc</td>
+<td>regproc</td>
+<td>pg_proc.oid</td>
+<td>The OID in <code class="ph codeph">pg_proc</code> of a function to compute the final value of a deferred-type window function from the partition row count and the result of <code class="ph codeph">winprefunc</code>.</td>
+</tr>
+<tr class="even">
+<td>winkind</td>
+<td>char</td>
+<td> </td>
+<td>A character indicating membership of the window function in a class of related functions:
+<p><code class="ph codeph">w</code> - ordinary window functions</p>
+<p><code class="ph codeph">n</code> - NTILE functions</p>
+<p><code class="ph codeph">f</code> - FIRST_VALUE functions</p>
+<p><code class="ph codeph">l</code> - LAST_VALUE functions</p>
+<p><code class="ph codeph">g</code> - LAG functions</p>
+<p><code class="ph codeph">d</code> - LEAD functions</p></td>
+</tr>
+</tbody>
+</table>
+
+
+
+
diff --git a/reference/cli/admin_utilities/analyzedb.html.md.erb b/reference/cli/admin_utilities/analyzedb.html.md.erb
new file mode 100644
index 0000000..429fb2d
--- /dev/null
+++ b/reference/cli/admin_utilities/analyzedb.html.md.erb
@@ -0,0 +1,141 @@
+---
+title: analyzedb
+---
+
+A utility that performs `ANALYZE` operations on tables incrementally and concurrently.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+analyzedb -d dbname
+ { -s schema |
+ { -t schema.table
+ [ -i col1[, col2, ...] |
+ -x col1[, col2, ...] ] } |
+ { -f | --file} config-file }
+ [ -l | --list ]
+ [ -p parallel-level ]
+ [ --full ]
+ [ --verbose ]
+ [ -a ]
+
+analyzedb { --clean_last | --clean_all }
+analyzedb --version
+analyzedb { -? | -h | --help }
+```
+
+## Description<a id="topic1__section3"></a>
+
+The `analyzedb` utility updates statistics on table data for the specified tables in a HAWQ database incrementally and concurrently.
+
+While performing `ANALYZE` operations, `analyzedb` creates a snapshot of the table metadata and stores it on disk on the master host. An `ANALYZE` operation is performed only if the table has been modified. If a table or partition has not been modified since the last time it was analyzed, `analyzedb` automatically skips the table or partition because it already contains up-to-date statistics.
+
+For a partitioned table `analyzedb` analyzes only those partitions that have no statistics, or that have stale statistics. `analyzedb` also refreshes the statistics on the root partition.
+
+By default, `analyzedb` creates a maximum of 5 concurrent sessions to analyze tables in parallel. For each session, `analyzedb` issues an `ANALYZE` command to the database and specifies different table names. The `-p` option controls the maximum number of concurrent sessions.
+
+## Notes<a id="topic1__section4"></a>
+
+The utility determines if a table has been modified by comparing catalog metadata of tables with the snapshot of metadata taken during a previous `analyzedb` operation. The snapshots of table metadata are stored as state files in the directory `db_analyze` in the HAWQ master data directory. You can specify the `--clean_last` or `--clean_all` option to remove state files generated by `analyzedb`.
+
+If you do not specify a table, set of tables, or schema, the `analyzedb` utility collects the statistics as needed on all system catalog tables and user-defined tables in the database.
+
+External tables are not affected by `analyzedb`.
+
+Table names that contain spaces are not supported.
+
+## Options<a id="topic1__section5"></a>
+
+--clean\_last
+Remove the state files generated by last `analyzedb` operation. All other options except `-d` are ignored.
+
+--clean\_all
+Remove all the state files generated by `analyzedb`. All other options except` -d` are ignored.
+
+-d *dbname*
+Specifies the name of the database that contains the tables to be analyzed. If this option is not specified, the database name is read from the environment variable `PGDATABASE`. If `PGDATABASE` is not set, the user name specified for the connection is used.
+
+-f *config-file* | --file *config-file*
+Text file that contains a list of tables to be analyzed. A relative file path from current directory can be specified.
+
+The file lists one table per line. Table names must be qualified with a schema name. Optionally, a list of columns can be specified using the `-i` or `-x`. No other options are allowed in the file. Other options such as `--full` must be specified on the command line.
+
+Only one of the options can be used to specify the files to be analyzed: `-f` or `--file`, `-t` , or `-s`.
+
+When performing `ANALYZE` operations on multiple tables, `analyzedb` creates concurrent sessions to analyze tables in parallel. The `-p` option controls the maximum number of concurrent sessions.
+
+In the following example, the first line performs an `ANALYZE` operation on the table `public.nation`, the second line performs an `ANALYZE` operation only on the columns `l_shipdate` and `l_receiptdate` in the table `public.lineitem`.
+
+``` pre
+public.nation
+public.lineitem -i l_shipdate, l_receiptdate
+```
+
+--full
+Perform an `ANALYZE` operation on all the specified tables. The operation is performed even if the statistics are up to date.
+
+-i *col1*, *col2*, ...
+Optional. Must be specified with the `-t` option. For the table specified with the `-t` option, collect statistics only for the specified columns.
+
+Only `-i`, or `-x` can be specified. Both options cannot be specified.
+
+-l | --list
+Lists the tables that would have been analyzed with the specified options. The `ANALYZE` operations are not performed.
+
+-p *parallel-level*
+The number of tables that are analyzed in parallel. *parallel level* can be an integer between 1 and 10, inclusive. Default value is 5.
+
+-s *schema*
+Specify a schema to analyze. All tables in the schema will be analyzed. Only a single schema name can be specified on the command line.
+
+Only one of the options can be used to specify the files to be analyzed: `-f` or `--file`, `-t` , or `-s`.
+
+-t *schema*.*table*
+Collect statistics only on *schema*.*table*. The table name must be qualified with a schema name. Only a single table name can be specified on the command line. You can specify the `-f` option to specify multiple tables in a file or the `-s` option to specify all the tables in a schema.
+
+Only one of these options can be used to specify the files to be analyzed: `-f` or `--file`, `-t` , or `-s`.
+
+-x *col1*, *col2*, ...
+Optional. Must be specified with the `-t` option. For the table specified with the `-t` option, exclude statistics collection for the specified columns. Statistics are collected only on the columns that are not listed.
+
+Only `-i`, or `-x` can be specified. Both options cannot be specified.
+
+-a
+Quiet mode. Do not prompt for user confirmation.
+
+-h | -? | --help
+Displays the online help.
+
+ -v | --verbose
+If specified, sets the logging level to verbose. Additional log information is written to the log file and the command line during command execution.
+
+--version
+Displays the version of this utility.
+
+## Examples<a id="topic1__section6"></a>
+
+An example that collects statistics only on a set of table columns. In the database `mytest`, collect statistics on the columns `shipdate` and `receiptdate` in the table `public.orders`:
+
+``` pre
+analyzedb -d mytest -t public.orders -i shipdate, receiptdate
+```
+
+An example that collects statistics on a table and exclude a set of columns. In the database `mytest`, collect statistics on the table `public.foo`, and do not collect statistics on the columns `bar` and `test2`.
+
+``` pre
+analyzedb -d mytest -t public.foo -x bar, test2
+```
+
+An example that specifies a file that contains a list of tables. This command collect statistics on the tables listed in the file `analyze-tables` in the database named `mytest`.
+
+``` pre
+analyzedb -d mytest -f analyze-tables
+```
+
+If you do not specify a table, set of tables, or schema, the `analyzedb` utility collects the statistics as needed on all catalog tables and user-defined tables in the specified database. This command refreshes table statistics on the system catalog tables and user-defined tables in the database `mytest`.
+
+``` pre
+analyzedb -d mytest
+```
+
+
diff --git a/reference/cli/admin_utilities/gpfdist.html.md.erb b/reference/cli/admin_utilities/gpfdist.html.md.erb
new file mode 100644
index 0000000..f1bd2b7
--- /dev/null
+++ b/reference/cli/admin_utilities/gpfdist.html.md.erb
@@ -0,0 +1,156 @@
+---
+title: gpfdist
+---
+
+Serves data files to or writes data files out from HAWQ segments.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+gpfdist [-d directory] [-p http_port] [-l log_file] [-t timeout]
+ [-S] [-w time] [-v | -V] [-s] [-m max_length] [--ssl certificate_path]
+
+gpfdist -? | --help
+
+gpfdist --version
+```
+
+## Description<a id="topic1__section3"></a>
+
+`gpfdist` is HAWQ parallel file distribution program. It is used by readable external tables and `hawq load` to serve external table files to all HAWQ segments in parallel. It is used by writable external tables to accept output streams from HAWQ segments in parallel and write them out to a file.
+
+In order for `gpfdist` to be used by an external table, the `LOCATION` clause of the external table definition must specify the external table data using the `gpfdist://` protocol (see the HAWQ command `CREATE EXTERNAL TABLE`).
+
+**Note:** If the `--ssl` option is specified to enable SSL security, create the external table with the `gpfdists://` protocol.
+
+The benefit of using `gpfdist` is that you are guaranteed maximum parallelism while reading from or writing to external tables, thereby offering the best performance as well as easier administration of external tables.
+
+For readable external tables, `gpfdist` parses and serves data files evenly to all the segment instances in the HAWQ system when users `SELECT` from the external table. For writable external tables, `gpfdist` accepts parallel output streams from the segments when users `INSERT` into the external table, and writes to an output file.
+
+For readable external tables, if load files are compressed using `gzip` or `bzip2` (have a `.gz` or `.bz2` file extension), `gpfdist` uncompresses the files automatically before loading provided that `gunzip` or `bunzip2` is in your path.
+
+**Note:** Currently, readable external tables do not support compression on Windows platforms, and writable external tables do not support compression on any platforms.
+
+Most likely, you will want to run `gpfdist` on your ETL machines rather than the hosts where HAWQ is installed. To install `gpfdist` on another host, simply copy the utility over to that host and add `gpfdist` to your `$PATH`.
+
+**Note:** When using IPv6, always enclose the numeric IP address in brackets.
+
+You can also run `gpfdist` as a Windows Service. See [Running gpfdist as a Windows Service](#topic1__section5) for more details.
+
+## Options<a id="topic1__section4"></a>
+
+-d *directory*
+The directory from which `gpfdist` will serve files for readable external tables or create output files for writable external tables. If not specified, defaults to the current directory.
+
+-l *log\_file*
+The fully qualified path and log file name where standard output messages are to be logged.
+
+-p *http\_port*
+The HTTP port on which `gpfdist` will serve files. Defaults to 8080.
+
+-t *timeout*
+Sets the time allowed for HAWQ to establish a connection to a `gpfdist` process. Default is 5 seconds. Allowed values are 2 to 600 seconds. May need to be increased on systems with a lot of network traffic.
+
+-m *max\_length*
+Sets the maximum allowed data row length in bytes. Default is 32768. Should be used when user data includes very wide rows (or when `line too long` error message occurs). Should not be used otherwise as it increases resource allocation. Valid range is 32K to 256MB. (The upper limit is 1MB on Windows systems.)
+
+-s
+Enables simplified logging. When this option is specified, only messages with `WARN` level and higher are written to the `gpfdist` log file. `INFO` level messages are not written to the log file. If this option is not specified, all `gpfdist` messages are written to the log file.
+
+You can specify this option to reduce the information written to the log file.
+
+-S (use O\_SYNC)
+Opens the file for synchronous I/O with the `O_SYNC` flag. Any writes to the resulting file descriptor block `gpfdist` until the data is physically written to the underlying hardware.
+
+-w *time*
+Sets the number of seconds that HAWQ delays before closing a target file such as a named pipe. The default value is 0, no delay. The maximum value is 600 seconds, 10 minutes.
+
+For a HAWQ with multiple segments, there might be a delay between segments when writing data from different segments to the file. You can specify a time to wait before HAWQ closes the file to ensure all the data is written to the file.
+
+--ssl *certificate\_path*
+Adds SSL encryption to data transferred with `gpfdist`. After executing `gpfdist` with the `--ssl certificate_path` option, the only way to load data from this file server is with the `gpfdist://` protocol.
+
+The location specified in *certificate\_path* must contain the following files:
+
+- The server certificate file, `server.crt`
+- The server private key file, `server.key`
+- The trusted certificate authorities, `root.crt`
+
+The root directory (`/`) cannot be specified as *certificate\_path*.
+
+-v (verbose)
+Verbose mode shows progress and status messages.
+
+-V (very verbose)
+Verbose mode shows all output messages generated by this utility.
+
+-? (help)
+Displays the online help.
+
+--version
+Displays the version of this utility.
+
+## Running gpfdist as a Windows Service<a id="topic1__section5"></a>
+
+HAWQ Loaders allow `gpfdist` to run as a Windows Service.
+
+Follow the instructions below to download, register and activate `gpfdist` as a service:
+
+1. Update your HAWQ Loader package to the latest version. This package is available from [Pivotal Network](https://network.gopivotal.com/products).
+2. Register `gpfdist` as a Windows service:
+ 1. Open a Windows command window
+ 2. Run the following command:
+
+ ``` pre
+ sc create gpfdist binpath= "path_to_gpfdist.exe -p 8081 -d External\load\files\path -l Log\file\path"
+ ```
+
+ You can create multiple instances of `gpfdist` by running the same command again, with a unique name and port number for each instance:
+
+ ``` pre
+ sc create gpfdistN binpath= "path_to_gpfdist.exe -p 8082 -d External\load\files\path -l Log\file\path"
+ ```
+
+3. Activate the `gpfdist` service:
+ 1. Open the Windows Control Panel and select **Administrative Tools > Services**.
+ 2. Highlight then right-click on the `gpfdist` service in the list of services.
+ 3. Select **Properties** from the right-click menu, the Service Properties window opens.
+
+ Note that you can also stop this service from the Service Properties window.
+
+ 4. Optional: Change the **Startup Type** to **Automatic** (after a system restart, this service will be running), then under **Service** status, click **Start**.
+ 5. Click **OK**.
+
+Repeat the above steps for each instance of `gpfdist` that you created.
+
+## Examples<a id="topic1__section6"></a>
+
+To serve files from a specified directory using port 8081 (and start `gpfdist` in the background):
+
+``` pre
+gpfdist -d /var/load_files -p 8081 &
+```
+
+To start `gpfdist` in the background and redirect output and errors to a log file:
+
+``` pre
+gpfdist -d /var/load_files -p 8081 -l /home/gpadmin/log &
+```
+
+To stop `gpfdist` when it is running in the background:
+
+--First find its process id:
+
+``` pre
+ps ax | grep gpfdist
+```
+
+--Then kill the process, for example:
+
+``` pre
+kill 3456
+```
+
+## See Also<a id="topic1__section7"></a>
+
+[hawq load](hawqload.html#topic1), [CREATE EXTERNAL TABLE](../../sql/CREATE-EXTERNAL-TABLE.html)
diff --git a/reference/cli/admin_utilities/gplogfilter.html.md.erb b/reference/cli/admin_utilities/gplogfilter.html.md.erb
new file mode 100644
index 0000000..2269ed8
--- /dev/null
+++ b/reference/cli/admin_utilities/gplogfilter.html.md.erb
@@ -0,0 +1,146 @@
+---
+title: gplogfilter
+---
+
+Searches through HAWQ log files for specified entries.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+gplogfilter [timestamp_options] [pattern_options]
+ [output_options] [input_options] [input_file]
+
+gplogfilter --help
+
+gplogfilter --version
+```
+
+## Description<a id="topic1__section3"></a>
+
+The `gplogfilter` utility can be used to search through a HAWQ log file for entries matching the specified criteria. If an input file is not supplied, then `gplogfilter` will use the `$MASTER_DATA_DIRECTORY` environment variable to locate the HAWQ master log file in the standard logging location. To read from standard input, use a dash (`-`) as the input file name. Input files may be compressed using `gzip`. In an input file, a log entry is identified by its timestamp in `YYYY-MM-DD [hh:mm[:ss]]` format.
+
+You can also use `gplogfilter` to search through all segment log files at once by running it through the [hawq ssh](hawqssh.html#topic1) utility. For example, to display the last three lines of each segment log file:
+
+``` pre
+hawq ssh -f seg_hostfile_hawqssh
+=> source ~/greenplum_path.sh
+=> gplogfilter -n 3 /data/hawq-install-path/segmentdd/pg_log/hawq*.csv
+```
+
+By default, the output of `gplogfilter` is sent to standard output. Use the `-o` option to send the output to a file or a directory. If you supply an output file name ending in `.gz`, the output file will be compressed by default using maximum compression. If the output destination is a directory, the output file is given the same name as the input file.
+
+## Options<a id="topic1__section4"></a>
+
+**Timestamp Options**
+
+-b *datetime* | --begin=*datetime*
+Specifies a starting date and time to begin searching for log entries in the format of `YYYY-MM-DD [hh:mm[:ss]]`.
+
+If a time is specified, the date and time must be enclosed in either single or double quotes. This example encloses the date and time in single quotes:
+
+``` pre
+gplogfilter -b '2016-02-13 14:23'
+```
+
+-e *datetime* | --end=*datetime*
+Specifies an ending date and time to stop searching for log entries in the format of `YYYY-MM-DD [hh:mm[:ss]]`.
+
+If a time is specified, the date and time must be enclosed in either single or double quotes. This example encloses the date and time in single quotes:
+
+``` pre
+gplogfilter -e '2016-02-13 14:23'
+```
+
+-d *time* | --duration=*time*
+Specifies a time duration to search for log entries in the format of `[hh][:mm[:ss]]`. If used without either the `-b` or `-e` option, will use the current time as a basis.
+
+**Pattern Matching Options**
+
+-c i \[gnore\] | r \[espect\] | --case=i \[gnore\] | r \[espect\]
+Matching of alphabetic characters is case sensitive by default unless proceeded by the `--case=ignore` option.
+
+-C '*string*' | --columns='*string*'
+Selects specific columns from the log file. Specify the desired columns as a comma-delimited string of column numbers beginning with 1, where the second column from left is 2, the third is 3, and so on.
+
+-f '*string*' | --find='*string*'
+Finds the log entries containing the specified string.
+
+-F '*string*' | --nofind='*string*'
+Rejects the log entries containing the specified string.
+
+-m *regex* | --match=*regex*
+Finds log entries that match the specified Python regular expression. See [http://docs.python.org/library/re.html](http://docs.python.org/library/re.html) for Python regular expression syntax.
+
+-M *regex* | --nomatch=*regex*
+Rejects log entries that match the specified Python regular expression. See [http://docs.python.org/library/re.html](http://docs.python.org/library/re.html) for Python regular expression syntax.
+
+-t | --trouble
+Finds only the log entries that have `ERROR:`, `FATAL:`, or `PANIC:` in the first line.
+
+**Output Options**
+
+-n *integer* | --tail=*integer*
+Limits the output to the last *integer* of qualifying log entries found.
+
+-s *offset* \[limit\] | --slice=*offset* \[limit\]
+From the list of qualifying log entries, returns the *limit* number of entries starting at the *offset* entry number, where an *offset* of zero (`0`) denotes the first entry in the result set and an *offset* of any number greater than zero counts back from the end of the result set.
+
+-o *output\_file* | --out=*output\_file*
+Writes the output to the specified file or directory location instead of `STDOUT`.
+
+-z 0-9 | --zip=0-9
+Compresses the output file to the specified compression level using `gzip`, where `0` is no compression and `9` is maximum compression. If you supply an output file name ending in `.gz`, the output file will be compressed by default using maximum compression.
+
+-a | --append
+If the output file already exists, appends to the file instead of overwriting it.
+
+**Input Options**
+
+input\_file
+The name of the input log file(s) to search through. If an input file is not supplied, `gplogfilter` will use the `$MASTER_DATA_DIRECTORY` environment variable to locate the HAWQ master log file. To read from standard input, use a dash (`-`) as the input file name.
+
+-u | --unzip
+Uncompress the input file using `gunzip`. If the input file name ends in `.gz`, it will be uncompressed by default.
+
+--help
+Displays the online help.
+
+--version
+Displays the version of this utility.
+
+## Examples<a id="topic1__section9"></a>
+
+Display the last three error messages in the master log file:
+
+``` pre
+gplogfilter -t -n 3
+```
+
+Display the last five error messages in a date-specified log file:
+
+``` pre
+gplogfilter -t -n 5 "/data/hawq-file-path/hawq-yyyy-mm-dd*.csv"
+```
+
+Display all log messages in the master log file timestamped in the last 10 minutes:
+
+``` pre
+gplogfilter -d :10
+```
+
+Display log messages in the master log file containing the string `|con6 cmd11|`:
+
+``` pre
+gplogfilter -f '|con6 cmd11|'
+```
+
+Using [hawq ssh](hawqssh.html#topic1), run `gplogfilter` on the segment hosts and search for log messages in the segment log files containing the string `con6` and save output to a file.
+
+``` pre
+hawq ssh -f /data/hawq-2.x/segmentdd/pg_hba.conf -e 'source /data/hawq-2.x/greenplum_path.sh ;
+gplogfilter -f con6 /data/hawq-2.x/pg_log/hawq*.csv' > seglog.out
+```
+
+## See Also<a id="topic1__section10"></a>
+
+[hawq ssh](hawqssh.html#topic1)
diff --git a/reference/cli/admin_utilities/gppkg.html.md.erb b/reference/cli/admin_utilities/gppkg.html.md.erb
new file mode 100644
index 0000000..1bfb6c7
--- /dev/null
+++ b/reference/cli/admin_utilities/gppkg.html.md.erb
@@ -0,0 +1,93 @@
+---
+title: gppkg
+---
+
+Installs HAWQ extensions such as pgcrypto, PL/R, PL/Java, and MADlib, along with their dependencies across an entire cluster.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+gppkg [-i package | -u package | -r name-version | -c]
+ [-d master_data_directory] [-a] [-v]
+
+gppkg --migrate GPHOME_1 GPHOME_2 [-a] [-v]
+
+gppkg [-q | --query] query_option
+
+gppkg -? | --help | -h
+
+gppkg --version
+```
+
+## Description<a id="topic1__section3"></a>
+
+The Package Manager (`gppkg`) utility installs HAWQ extensions, along with any dependencies, on all hosts across a cluster.
+
+First, download one or more of the available packages from [Pivotal Network](https://network.gopivotal.com/products); then copy it to the master host. Use the Package Manager to install each package using the options described below.
+
+**Note:** After a major upgrade to HAWQ, you must download and install all extensions again.
+
+Examples of database extensions and packages software that are delivered using the Package Manager are:
+
+- PL/Java
+- PL/R
+- PL/Perl
+- MADlib
+- pgcrypto
+
+Note that Package Manager installation files for extension packages might release outside of standard product release cycles. Current packages are available from [Pivotal Network](https://network.gopivotal.com/products).
+
+## Options<a id="topic1__section4"></a>
+
+-a (do not prompt)
+Do not prompt the user for confirmation.
+
+-c | --clean
+Reconciles the package state of the cluster to match the state of the master host. Running this option after a failed or partial install/uninstall ensures that the package installation state is consistent across the cluster.
+
+-d *master\_data\_directory*
+The master data directory. If not specified, the value set for `$MASTER_DATA_DIRECTORY` will be used.
+
+-i *package* | --install=*package*
+Installs the given package. This includes any pre/post installation steps and installation of any dependencies.
+
+--migrate *GPHOME\_1* *GPHOME\_2*
+Migrates packages from a separate `$GPHOME`. Carries over packages from one version of HAWQ to another.
+
+For example: `gppkg --migrate /usr/local/hawq-old-version /usr/local/hawq-new-version`
+
+This option is automatically invoked by the installer during minor upgrades. This option is given here for cases when the user wants to migrate packages manually.
+
+Migration can only proceed if `gppkg` is executed from the installation directory to which packages are being migrated. That is, `GPHOME_2` must match the `$GPHOME` from which the currently executing `gppkg` is being run.
+
+-q | --query *query\_option*
+Provides information specified by `query_option` about the installed packages. Only one `query_option` can be specified at a time. The following table lists the possible values for query\_option. `<package_file>` is the name of a package. <a id="topic1__jt142509"></a>
+
+<span class="tablecap">Table 1. Query Options for gppkg</span>
+
+| query\_option | Returns |
+|-------------------------|-----------------------------------------------------------------------|
+| `<package_file>` | Whether the specified package is installed. |
+| `--info <package_file>` | The name, version, and other information about the specified package. |
+| `--list <package_file>` | The file contents of the specified package. |
+| `--all` | List of all installed packages. |
+
+
+-r *name-version* | --remove=*name-version*
+Removes the specified package.
+
+-u *package* | --update=*package*
+Updates the given package.
+
+**Warning:** The process of updating a package includes removing all previous versions of the system objects related to the package. For example, previous versions of shared libraries are removed. After the update process, a database function will fail when it is called if the function references a package file that has been removed.
+
+--version (show utility version)
+Displays the version of this utility.
+
+-v | --verbose
+Sets the logging level to verbose.
+
+-? | -h | --help
+Displays the online help.
+
+
diff --git a/reference/cli/admin_utilities/hawqactivate.html.md.erb b/reference/cli/admin_utilities/hawqactivate.html.md.erb
new file mode 100644
index 0000000..f807228
--- /dev/null
+++ b/reference/cli/admin_utilities/hawqactivate.html.md.erb
@@ -0,0 +1,83 @@
+---
+title: hawq activate
+---
+
+Activates a standby master host and makes it the active master for the HAWQ system.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+hawq activate standby [-t time | --timeout time]
+ [-l logfile_directory]
+hawq activate [-M [smart|fast|immediate] --mode [smart|fast|immediate] ]
+ [--ignore_bad-hosts]
+hawq activate -v | [-q]
+
+hawq activate -? | -h | --help
+```
+
+## Description<a id="topic1__section3"></a>
+
+If the primary master fails, the log replication process is shut down, and the standby master can be activated in its place. The `hawq activate standby` utility activates a backup, standby master host and brings it into operation as the active master instance for a HAWQ system. The activated standby master effectively becomes the HAWQ master, accepting client connections on the master port.
+
+When you initialize a standby master, the default is to use the same port as the active master. For information about the master port for the standby master, see [hawq init](hawqinit.html#topic1).
+
+You must run this utility from the master host you are activating, not the failed master host you are disabling. Running this utility assumes you have a standby master host configured for the system .
+
+The utility will perform the following steps:
+
+- Stops the synchronization process (`walreceiver`) on the standby master
+- Updates the system catalog tables of the standby master using the logs
+- Activates the standby master to be the new active master for the system
+- Restarts the HAWQ system with the new master host
+
+In order to use `hawq activate standby` to activate a new primary master host, the master host that was previously serving as the primary master cannot be running. The utility checks for a `postmaster.pid` file in the data directory of the disabled master host, and if it finds it there, it will assume the old master host is still active. In some cases, you may need to remove the `postmaster.pid` file from the disabled master host data directory before running `hawq activate standby` (for example, if the disabled master host process was terminated unexpectedly).
+
+After activating a standby master, run `ANALYZE` to update the database query statistics. For example:
+
+``` pre
+psql dbname -c 'ANALYZE;'
+```
+
+After you activate the standby master as the primary master, the HAWQ system no longer has a standby master configured. You might want to specify another host to be the new standby with the [hawq init](hawqinit.html#topic1) utility.
+
+## Options<a id="topic1__section4"></a>
+
+-l --logdir *logfile\_directory*
+Specifies the log directory for logs of the management tools. The default is `~/hawq/Adminlogs/`.
+
+-t --timeout *timeout\_seconds*
+Seconds to wait before discontinuing the operation. If not specified, the default timeout is 60 seconds.
+
+-M --mode smart | fast | immediate
+Stop with one of the specified modes.
+
+Smart shutdown is the default. Shutdown fails with a warning message, if active connections are found.
+
+Fast shut down interrupts and rolls back any transactions currently in progress .
+
+Immediate shutdown aborts transactions in progress and kills all `postgres` processes without allowing the database server to complete transaction processing or clean up any temporary or in-process work files. Because of this, immediate shutdown is not recommended. In some instances, it can cause database corruption that requires manual recovery.
+
+--ignore\_bad\_hosts
+Overrides copying configuration files to a host on which SSH validation fails. If ssh to a skipped host is reestablished, make sure the configuration files are re-synched once it is reachable.
+
+-q (no screen output)
+Run in quiet mode. Command output is not displayed on the screen, but is still written to the log file.
+
+ -v --verbose
+Displays detailed status, progress and error messages output by the utility.
+
+-? | -h | --help (help)
+Displays the online help.
+
+## Example<a id="topic1__section5"></a>
+
+Activate the standby master host and make it the active master instance for a HAWQ system (run from backup master host you are activating):
+
+``` pre
+hawq activate standby /gpdata
+```
+
+## See Also<a id="topic1__section6"></a>
+
+[hawq init](hawqinit.html#topic1)
diff --git a/reference/cli/admin_utilities/hawqcheck.html.md.erb b/reference/cli/admin_utilities/hawqcheck.html.md.erb
new file mode 100644
index 0000000..ebb6991
--- /dev/null
+++ b/reference/cli/admin_utilities/hawqcheck.html.md.erb
@@ -0,0 +1,120 @@
+---
+title: hawq check
+---
+
+Verifies and validates HAWQ platform settings.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+hawq check -f hostfile_hawq_check [--hadoop | --hadoop hadoop_home]
+ [--stdout | --zipout] [--config config_file]
+ [--kerberos] [--hdfs-ha] [--yarn] [--yarn-ha]
+hawq check --zipin hawq_check_zipfile
+
+hawq check -d temp_directory
+ {-f hostfile_checknet | - h hostname [-h hostname ...]}
+ [ -r n|N|M [--duration time] [--netperf] ] [-D] [-v | -V]
+
+hawq check -?
+
+hawq check --version
+```
+
+## Description<a id="topic1__section3"></a>
+
+The `hawq check` utility determines the platform on which you are running HAWQ and validates various platform-specific configuration settings as well as HAWQ and HDFS-specific configuration settings. In order to perform HAWQ configuration checks, make sure HAWQ has been already started and `hawq config` works. For HDFS checks, you should either set the HADOOP\_HOME environment variable or give the hadoop installation location using `--hadoop` option.
+
+The `hawq check` utility can use a host file or a file previously created with the `--zipout `option to validate platform settings. If `GPCHECK_ERROR` displays, one or more validation checks failed. You can also use `hawq check` to gather and view platform settings on hosts without running validation checks. When running checks, `hawq check` compares your actual configuration setting with an expected value listed in a config file (`$GPHOME/etc/hawq check.cnf` by default). You must modify your configuration values for "mount.points" and "diskusage.monitor.mounts" to reflect the actual mount points you want to check, as a comma-separated list. Otherwise, the utility only checks the root directory, which may not be helpful.
+
+An example is shown below:
+
+``` pre
+[linux.mount]
+mount.points = /,/data1,/data2
+
+[linux.diskusage]
+diskusage.monitor.mounts = /,/data1,/data2
+```
+
+## Options<a id="topic1__section4"></a>
+
+--config *config\_file*
+The name of a configuration file to use instead of the default file `$GPHOME/etc/hawq_check.cnf`.
+
+-f *hostfile\_hawq\_check*
+The name of a file that contains a list of hosts that `hawq check` uses to validate platform-specific settings. This file should contain a single host name for all hosts in your HAWQ system (master, standby master, and segments).
+
+--hadoop | --hadoop-home *hadoop\_home*
+Use this option to specify your hadoop installation location so that `hawq check` can validate HDFS settings. This option is not needed if `HADOOP_HOME` environment variable is set.
+
+--kerberos
+Use this option to check HDFS and YARN when running Kerberos mode. This allows `hawq check` to validate HAWQ/HDFS/YARN settings with Kerberos enabled.
+
+--hdfs-ha
+Use this option to indicate that HDFS-HA mode is enabled, allowing `hawq check` to validate HDFS settings with HA mode enabled.
+
+--yarn
+If HAWQ is using YARN, enables yarn mode, allowing `hawq check` to validate the basic YARN settings.
+
+--yarn-ha
+Use this option to indicate HAWQ is using YARN with High Availability mode enabled, to allow `hawq check` to validate HAWQ-YARN settings with YARN-HA enabled.
+
+--stdout
+Send collected host information from `hawq check` to standard output. No checks or validations are performed.
+
+--zipout
+Save all collected data to a .zip file in the current working directory. `hawq check` automatically creates the .zip file and names it `hawq_check_timestamp.tar.gz.` No checks or validations are performed.
+
+--zipin *file*
+Use this option to decompress and check a .zip file created with the `--zipout` option. If you specify the `--zipin` option, `hawq check` performs validation tasks against the specified file.
+
+--version
+Displays the version of this utility.
+
+-? (help)
+Displays the online help.
+
+## Examples<a id="topic1__section5"></a>
+
+Verify and validate the HAWQ platform settings by entering a host file and specifying the hadoop location:
+
+``` pre
+# hawq check -f hostfile_hawq_check --hadoop ~/hadoop-2.0.0/
+```
+
+Verify and validate the HAWQ platform settings with HDFS HA enabled, YARN HA enabled and Kerberos enabled:
+
+``` pre
+# hawq check -f hostfile_hawq_check --hadoop ~/hadoop-2.0.0/
+ --hdfs-ha --yarn-ha --kerberos
+```
+
+Verify and validate the HAWQ platform settings with HDFS HA enabled, and Kerberos enabled:
+
+``` pre
+# hawq check -f hostfile_hawq_check --hadoop ~/hadoop-2.0.0/
+ --hdfs-ha --kerberos
+```
+
+Save HAWQ platform settings to a zip file, when HADOOP\_HOME environment variable is set:
+
+``` pre
+# hawq check -f hostfile_hawq_check --zipout
+```
+
+Verify and validate the HAWQ platform settings using a zip file created with the --zipout option:
+
+``` pre
+# hawq check --zipin hawq_check_timestamp.tar.gz
+```
+
+View collected HAWQ platform settings:
+
+``` pre
+# hawq check -f hostfile_hawq_check --hadoop ~/hadoop-2.0.0/ --stdout
+```
+
+## See Also<a id="topic1__section6"></a>
+
+[hawq checkperf](hawqcheckperf.html#topic1)
diff --git a/reference/cli/admin_utilities/hawqcheckperf.html.md.erb b/reference/cli/admin_utilities/hawqcheckperf.html.md.erb
new file mode 100644
index 0000000..7a93b91
--- /dev/null
+++ b/reference/cli/admin_utilities/hawqcheckperf.html.md.erb
@@ -0,0 +1,127 @@
+---
+title: hawq checkperf
+---
+
+Verifies the baseline hardware performance of the specified hosts.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+hawq checkperf -d test_directory [-d test_directory ...]
+ {-f hostfile_checkperf | - h hostname [-h hostname ...]}
+ [-r ds] [-B block_size] [-S file_size] [-D] [-v|-V]
+
+hawq checkperf -d temp_directory
+ {-f hostfile_checknet | - h hostname [-h hostname ...]}
+ [ -r n|N|M [--duration time] [--netperf] ] [-D] [-v | -V]
+
+hawq checkperf -?
+
+hawq checkperf --version
+```
+
+## Description<a id="topic1__section3"></a>
+
+The `hawq checkperf` utility starts a session on the specified hosts and runs the following performance tests:
+
+- **Disk I/O Test (dd test)** — To test the sequential throughput performance of a logical disk or file system, the utility uses the **dd** command, which is a standard UNIX utility. It times how long it takes to write and read a large file to and from disk and calculates your disk I/O performance in megabytes (MB) per second. By default, the file size that is used for the test is calculated at two times the total random access memory (RAM) on the host. This ensures that the test is truly testing disk I/O and not using the memory cache.
+- **Memory Bandwidth Test (stream)** — To test memory bandwidth, the utility uses the STREAM benchmark program to measure sustainable memory bandwidth (in MB/s). This tests that your system is not limited in performance by the memory bandwidth of the system in relation to the computational performance of the CPU. In applications where the data set is large (as in HAWQ), low memory bandwidth is a major performance issue. If memory bandwidth is significantly lower than the theoretical bandwidth of the CPU, then it can cause the CPU to spend significant amounts of time waiting for data to arrive from system memory.
+- **Network Performance Test (gpnetbench\*)** — To test network performance (and thereby the performance of the HAWQ interconnect), the utility runs a network benchmark program that transfers a 5 second stream of data from the current host to each remote host included in the test. The data is transferred in parallel to each remote host and the minimum, maximum, average and median network transfer rates are reported in megabytes (MB) per second. If the summary transfer rate is slower than expected (less than 100 MB/s), you can run the network test serially using the `-r n` option to obtain per-host results. To run a full-matrix bandwidth test, you can specify `-r M` which will cause every host to send and receive data from every other host specified. This test is best used to validate if the switch fabric can tolerate a full-matrix workload.
+
+To specify the hosts to test, use the `-f` option to specify a file containing a list of host names, or use the `-h` option to name single host names on the command-line. If running the network performance test, all entries in the host file must be for network interfaces within the same subnet. If your segment hosts have multiple network interfaces configured on different subnets, run the network test once for each subnet.
+
+You must also specify at least one test directory (with `-d`). The user who runs `hawq checkperf` must have write access to the specified test directories on all remote hosts. For the disk I/O test, the test directories should correspond to your segment data directories. For the memory bandwidth and network tests, a temporary directory is required for the test program files.
+
+Before using `hawq checkperf`, you must have a trusted host setup between the hosts involved in the performance test. You can use the utility `hawq ssh-exkeys` to update the known host files and exchange public keys between hosts if you have not done so already. Note that `hawq checkperf` calls to `hawq ssh` and `hawq scp`, so these HAWQ utilities must also be in your `$PATH`.
+
+## Options<a id="topic1__section4"></a>
+
+-B *block\_size*
+Specifies the block size (in KB or MB) to use for disk I/O test. The default is 32KB, which is the same as the HAWQ page size. The maximum block size is 1 MB.
+
+-d *test\_directory*
+For the disk I/O test, specifies the file system directory locations to test. You must have write access to the test directory on all hosts involved in the performance test. You can use the `-d` option multiple times to specify multiple test directories (for example, to test disk I/O of your data directories).
+
+-d *temp\_directory*
+For the network and stream tests, specifies a single directory where the test program files will be copied for the duration of the test. You must have write access to this directory on all hosts involved in the test.
+
+-D (display per-host results)
+Reports performance results for each host for the disk I/O tests. The default is to report results for just the hosts with the minimum and maximum performance, as well as the total and average performance of all hosts.
+
+--duration *time*
+Specifies the duration of the network test in seconds (s), minutes (m), hours (h), or days (d). The default is 15 seconds.
+
+-f *hostfile\_checkperf*
+For the disk I/O and stream tests, specifies the name of a file that contains one host name per host that will participate in the performance test. The host name is required, and you can optionally specify an alternate user name and/or SSH port number per host. The syntax of the host file is one host per line as follows:
+
+``` pre
+[username@]hostname[:ssh_port]
+```
+
+-f *hostfile\_checknet*
+For the network performance test, all entries in the host file must be for host adresses within the same subnet. If your segment hosts have multiple network interfaces configured on different subnets, run the network test once for each subnet. For example (a host file containing segment host address names for interconnect subnet 1):
+
+``` pre
+sdw1-1
+sdw2-1
+sdw3-1
+```
+
+-h *hostname*
+Specifies a single host name (or host address) that will participate in the performance test. You can use the `-h` option multiple times to specify multiple host names.
+
+--netperf
+Specifies that the `netperf` binary should be used to perform the network test instead of the HAWQ network test. To use this option, you must download `netperf` from [http://www.netperf.org](http://www.netperf.org) and install it into `$GPHOME/bin/lib` on all HAWQ hosts (master and segments).
+
+-r ds{n|N|M}
+Specifies which performance tests to run. The default is `dsn`:
+
+- Disk I/O test (`d`)
+- Stream test (`s`)
+- Network performance test in sequential (`n`), parallel (`N`), or full-matrix (`M`) mode. The optional `--duration` option specifies how long (in seconds) to run the network test. To use the parallel (`N`) mode, you must run the test on an *even* number of hosts.
+
+ If you would rather use `netperf` ([http://www.netperf.org](http://www.netperf.org)) instead of the HAWQ network test, you can download it and install it into `$GPHOME/bin/lib` on all HAWQ hosts (master and segments). You would then specify the optional `--netperf` option to use the `netperf` binary instead of the default `gpnetbench*` utilities.
+
+-S *file\_size*
+Specifies the total file size to be used for the disk I/O test for all directories specified with `-d`. *file\_size* should equal two times total RAM on the host. If not specified, the default is calculated at two times the total RAM on the host where `hawq checkperf` is executed. This ensures that the test is truly testing disk I/O and not using the memory cache. You can specify sizing in KB, MB, or GB.
+
+-v (verbose) | -V (very verbose)
+Verbose mode shows progress and status messages of the performance tests as they are run. Very verbose mode shows all output messages generated by this utility.
+
+--version
+Displays the version of this utility.
+
+-? (help)
+Displays the online help.
+
+## Examples<a id="topic1__section5"></a>
+
+Run the disk I/O and memory bandwidth tests on all the hosts in the file *host\_file* using the test directory of */data1* and */data2*:
+
+``` pre
+$ hawq checkperf -f hostfile_checkperf -d /data1 -d /data2 -r ds
+```
+
+Run only the disk I/O test on the hosts named *sdw1* and sdw2 using the test directory of */data1*. Show individual host results and run in verbose mode:
+
+``` pre
+$ hawq checkperf -h sdw1 -h sdw2 -d /data1 -r d -D -v
+```
+
+Run the parallel network test using the test directory of */tmp,* where *hostfile\_check\_ic\** specifies all network interface host address names within the same interconnect subnet:
+
+``` pre
+$ hawq checkperf -f hostfile_checknet_ic1 -r N -d /tmp
+$ hawq checkperf -f hostfile_checknet_ic2 -r N -d /tmp
+```
+
+Run the same test as above, but use `netperf` instead of the HAWQ network test (note that `netperf` must be installed in `$GPHOME/bin/lib` on all HAWQ hosts):
+
+``` pre
+$ hawq checkperf -f hostfile_checknet_ic1 -r N --netperf -d /tmp
+$ hawq checkperf -f hostfile_checknet_ic2 -r N --netperf -d /tmp
+```
+
+## See Also<a id="topic1__section6"></a>
+
+[hawq ssh](hawqssh.html#topic1), [hawq scp](hawqscp.html#topic1)
diff --git a/reference/cli/admin_utilities/hawqconfig.html.md.erb b/reference/cli/admin_utilities/hawqconfig.html.md.erb
new file mode 100644
index 0000000..311a8ba
--- /dev/null
+++ b/reference/cli/admin_utilities/hawqconfig.html.md.erb
@@ -0,0 +1,99 @@
+---
+title: hawq config
+---
+
+Sets server configuration parameters on all nodes (master and segments) within a HAWQ system.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+hawq config [options]]
+```
+
+``` pre
+hawq config [-c hawq_property | --change=hawq_property]
+ | -r param_name --remove=hawq_property [ -l | --list]
+ [--skipvalidation] [--ignore_bad-hosts]
+ [-s hawq_property | --show hawq_property]
+ [-v hawq_property_value | --value=hawq_property_value] [-d hawq_home_directory]
+
+hawq config --help
+```
+
+## Description<a id="topic1__section3"></a>
+
+The `hawq config` utility allows you to set, unset, or view configuration properties from the `hawq-site.xml` files of all instances in your HAWQ system.
+
+**Note:** The `hawq config` utility makes configuration properties identical and consistent across all nodes, including the master and segments. Using the utility will override any unique configurations that were defined manually in `hawq-site.xml`.
+
+`hawq config` can only be used to manage specific properties. For example, you cannot use it to set properties such as `port`, which is required to be distinct for every segment instance. Use the `-l` (list) option to see a complete list of configuration properties supported by `hawq config`.
+
+When `hawq config` sets a configuration property in a `hawq_site.xml` file, the new property setting always displays at the bottom of the file. When you use `hawq config` to remove a configuration property setting, `hawq config` comments out the property in all `hawq-site.xml` files, thereby restoring the system default setting. For example, if you use `hawq config `to remove (comment out) a property and later add it back (set a new value), there will be two instances of the property; one that is commented out, and one that is enabled and inserted at the bottom of the `hawq-site.xml` file.
+
+After setting a property, you must restart your HAWQ system or reload the `hawq-site.xml` file for the change to take effect. Whether you require a restart or a reload depends on the property being set. To reload the configuration files, use `hawq stop -u`. To restart the system, use `hawq restart` .
+
+To show the currently set values for a property across the system, use the `-s` option.
+
+`hawq config` uses the following environment variables to connect to the HAWQ master instance and obtain system configuration information:
+
+- `PGHOST`
+- `PGPORT`
+- `PGUSER`
+- `PGPASSWORD`
+- `PGDATABASE`
+
+## Options<a id="topic1__section4"></a>
+
+-c | --change *hawq\_property*
+Changes a HAWQ property setting by adding the new setting to the bottom of the `hawq-site.xml` files.
+
+-r | --remove *hawq\_property*
+Removes a HAWQ property setting by commenting out the entry in the `hawq-site.xml` files.
+
+-l | --list
+Lists all HAWQ property settings supported by the `hawq config` utility.
+
+-s | --show *property\_name*
+Shows the value for a HAWQ property name used on all instances (master and segments) in the HAWQ system. If there is a discrepancy in a parameter value between segment instances, the `hawq config` utility displays an error message. Note that the `hawq config` utility reads property values directly from the database, and not the `hawq-site.xml` file. If you are using `hawq config` to set properties across all segments, then running `hawq config -s` to verify the changes, you might still see the previous (old) values. You must reload the configuration files (`hawq stop -u`) or restart the system (`hawq restart`) for changes to take effect.
+
+-v | --value *hproperty\_value*
+Set the value of the HAWQ property setting in the `hawq-site.xml` files.
+
+--skipvalidation
+Overrides the system validation checks of `hawq config` and allows you to operate on any server property, including hidden parameters and restricted parameters that cannot be changed by `hawq config`. Do not modify hidden or restricted parameters unless you are aware of all potential consequences. When used with the `-l` list option, it shows the list of restricted properties.
+
+--ignore\_bad\_hosts
+Overrides copying configuration files to a host on which SSH validation fails. If ssh to a skipped host is reestablished, make sure the configuration files are re-synched once it is reachable.
+
+-? | -h | --help
+Displays the online help.
+
+## Examples<a id="topic1__section5"></a>
+
+Set the `max_connections` setting to 100:
+
+``` pre
+hawq config -c max_connections -v 100
+```
+
+Comment out all instances of the `default_statistics_target` property, and restore the system default:
+
+``` pre
+hawq config -r default_statistics_target
+```
+
+List all properties supported by `hawq config`:
+
+``` pre
+hawq config -l
+```
+
+Show the values of a particular property across the system:
+
+``` pre
+hawq config -s max_connections
+```
+
+## See Also<a id="topic1__section6"></a>
+
+[hawq stop](hawqstop.html#topic1)
diff --git a/reference/cli/admin_utilities/hawqextract.html.md.erb b/reference/cli/admin_utilities/hawqextract.html.md.erb
new file mode 100644
index 0000000..ca09d15
--- /dev/null
+++ b/reference/cli/admin_utilities/hawqextract.html.md.erb
@@ -0,0 +1,288 @@
+---
+title: hawq extract
+---
+
+Extracts the metadata of a specified table into a YAML file.
+
+## Synopsis
+
+``` pre
+hawq extract [-h hostname] [-p port] [-U username] [-d database]
+ [-o output_file] [-W] <tablename>
+
+hawq extract -?
+
+hawq extract --version
+```
+
+## Description
+
+`hawq extract` is a utility that extracts a table's metadata into a YAML formatted file. HAWQ's InputFormat uses this YAML-formatted file to read a HAWQ file stored on HDFS directly into the MapReduce program.
+
+**Note:**
+`hawq extract` is bound by the following rules:
+
+- You must start up HAWQ to use `hawq extract`.
+- `hawq extract` only supports AO and Parquet tables.
+- `hawq extract` supports partitioned tables, but does not support sub-partitions.
+
+## Arguments
+
+<tablename>
+Name of the table that you need to extract metadata. You can use the format *namespace\_name.table\_name*.
+
+## Options
+
+-o *output\_file*
+Is the name of a file that `hawq extract` uses to write the metadata. If you do not specify a name, `hawq extract` writes to `stdout`.
+
+<!-- -->
+
+-v (verbose mode)
+Optional. Displays the verbose output of the extraction process.
+
+<!-- -->
+
+-? (help)
+Displays the online help.
+
+--version
+Displays the version of this utility.
+
+**Connection Options**
+
+-d *database*
+The database to connect to. If not specified, it reads from the environment variable `$PGDATABASE` or defaults to `template1`.
+
+-h *hostname*
+Specifies the host name of the machine on which the HAWQ master database server is running. If not specified, it reads from the environment variable `$PGHOST` or defaults to `localhost`.
+
+-p *port*
+Specifies the TCP port on which the HAWQ master database server is listening for connections. If not specified, reads from the environment variable `$PGPORT` or defaults to 5432.
+
+-U *username*
+The database role name to connect as. If not specified, reads from the environment variable `$PGUSER` or defaults to the current system user name.
+
+-W (force password prompt)
+Force a password prompt. If not specified, reads the password from the environment variable `$PGPASSWORD` or from a password file specified by `$PGPASSFILE` or in `~/.pgpass`.
+
+## Metadata File Format
+
+`hawq extract` exports the table metadata into a file using YAML 1.1 document format. The file contains various key information about the table, such as table schema, data file locations and sizes, partition constraints and so on.
+
+The basic structure of the metadata file is as follows:
+
+``` pre
+Version: string (1.0.0)
+DBVersion: string
+FileFormat: string (AO/Parquet)
+TableName: string (schemaname.tablename)
+DFS_URL: string (hdfs://127.0.0.1:9000)
+Encoding: UTF8
+AO_Schema:
+ - name: string
+ type: string
+
+AO_FileLocations:
+ Blocksize: int
+ Checksum: boolean
+ CompressionType: string
+ CompressionLevel: int
+ PartitionBy: string ('PARTITION BY ...')
+ Files:
+ - path: string (/gpseg0/16385/35469/35470.1)
+ size: long
+
+ Partitions:
+ - Blocksize: int
+ Checksum: Boolean
+ CompressionType: string
+ CompressionLevel: int
+ Name: string
+ Constraint: string (PARTITION Jan08 START (date '2008-01-01') INCLUSIVE)
+ Files:
+ - path: string
+ size: long
+
+
+Parquet_FileLocations:
+ RowGroupSize: long
+ PageSize: long
+ CompressionType: string
+ CompressionLevel: int
+ Checksum: boolean
+ EnableDictionary: boolean
+ PartitionBy: string
+ Files:
+ - path: string
+ size: long
+ Partitions:
+ - Name: string
+ RowGroupSize: long
+ PageSize: long
+ CompressionType: string
+ CompressionLevel: int
+ Checksum: boolean
+ EnableDictionary: boolean
+ Constraint: string
+ Files:
+ - path: string
+ size: long
+```
+
+## Example - Extracting an AO table
+
+Extract the `rank` table's metadata into a file named `rank_table.yaml`:
+
+``` pre
+$ hawq extract -o rank_table.yaml -d postgres rank
+```
+
+**Output content in rank\_table.yaml**
+
+``` pre
+AO_FileLocations:
+ Blocksize: 32768
+ Checksum: false
+ CompressionLevel: 0
+ CompressionType: null
+ Files:
+ - path: /gpseg0/16385/35469/35692.1
+ size: 0
+ - path: /gpseg1/16385/35469/35692.1
+ size: 0
+ PartitionBy: PARTITION BY list (gender)
+ Partitions:
+ - Blocksize: 32768
+ Checksum: false
+ CompressionLevel: 0
+ CompressionType: null
+ Constraint: PARTITION girls VALUES('F') WITH (appendonly=true)
+ Files:
+ - path: /gpseg0/16385/35469/35697.1
+ size: 0
+ - path: /gpseg1/16385/35469/35697.1
+ size: 0
+ Name: girls
+ - Blocksize: 32768
+ Checksum: false
+ CompressionLevel: 0
+ CompressionType: null
+ Constraint: PARTITION boys VALUES('M') WITH (appendonly=true)
+ Files:
+ - path: /gpseg0/16385/35469/35703.1
+ size: 0
+ - path: /gpseg1/16385/35469/35703.1
+ size: 0
+ Name: boys
+ - Blocksize: 32768
+ Checksum: false
+ CompressionLevel: 0
+ CompressionType: null
+ Constraint: DEFAULT PARTITION other WITH appendonly=true)
+ Files:
+ - path: /gpseg0/16385/35469/35709.1
+ size: 90071728
+ - path: /gpseg1/16385/35469/35709.1
+ size: 90071512
+ Name: other
+ AO_Schema:
+ - name: id
+ type: int4
+ - name: rank
+ type: int4
+ - name: year
+ type: int4
+ - name: gender
+ type: bpchar
+ - name: count
+ type: int4
+ DFS_URL: hdfs://127.0.0.1:9000
+ Encoding: UTF8
+ FileFormat: AO
+ TableName: public.rank
+ Version: 1.0.0
+```
+
+## Example - Extracting a Parquet table
+
+Extract the `orders` table's metadata into a file named `orders.yaml`:
+
+``` pre
+$ hawq extract -o orders.yaml -d postgres orders
+```
+
+**Output content in orders.yaml**
+
+``` pre
+DFS_URL: hdfs://127.0.0.1:9000
+Encoding: UTF8
+FileFormat: Parquet
+TableName: public.orders
+Version: 1.0.0
+Parquet_FileLocations:
+ Checksum: false
+ CompressionLevel: 0
+ CompressionType: none
+ EnableDictionary: false
+ Files:
+ - path: /hawq-data/gpseg0/16385/16626/16657.1
+ size: 0
+ - path: /hawq-data/gpseg1/16385/16626/16657.1
+ size: 0
+ PageSize: 1048576
+ PartitionBy: PARTITION BY range (o_orderdate)
+ Partitions:
+ - Checksum: false
+ CompressionLevel: 0
+ CompressionType: none
+ Constraint: PARTITION p1_1 START ('1992-01-01'::date) END ('1994-12-31'::date)
+ EVERY ('3 years'::interval) WITH (appendonly=true, orientation=parquet, pagesize=1048576,
+ rowgroupsize=8388608, compresstype=none, compresslevel=0)
+ EnableDictionary: false
+ Files:
+ - path: /hawq-data/gpseg0/16385/16626/16662.1
+ size: 8140599
+ - path: /hawq-data/gpseg1/16385/16626/16662.1
+ size: 8099760
+ Name: orders_1_prt_p1_1
+ PageSize: 1048576
+ RowGroupSize: 8388608
+ - Checksum: false
+ CompressionLevel: 0
+ CompressionType: none
+ Constraint: PARTITION p1_11 START ('1995-01-01'::date) END ('1997-12-31'::date)
+ EVERY ('e years'::interval) WITH (appendonly=true, orientation=parquet, pagesize=1048576,
+ rowgroupsize=8388608, compresstype=none, compresslevel=0)
+ EnableDictionary: false
+ Files:
+ - path: /hawq-data/gpseg0/16385/16626/16668.1
+ size: 8088559
+ - path: /hawq-data/gpseg1/16385/16626/16668.1
+ size: 8075056
+ Name: orders_1_prt_p1_11
+ PageSize: 1048576
+ RowGroupSize: 8388608
+ - Checksum: false
+ CompressionLevel: 0
+ CompressionType: none
+ Constraint: PARTITION p1_21 START ('1998-01-01'::date) END ('2000-12-31'::date)
+ EVERY ('3 years'::interval) WITH (appendonly=true, orientation=parquet, pagesize=1048576,
+ rowgroupsize=8388608, compresstype=none, compresslevel=0)
+ EnableDictionary: false
+ Files:
+ - path: /hawq-data/gpseg0/16385/16626/16674.1
+ size: 8065770
+ - path: /hawq-data/gpseg1/16385/16626/16674.1
+ size: 8126669
+ Name: orders_1_prt_p1_21
+ PageSize: 1048576
+ RowGroupSize: 8388608
+ RowGroupSize: 8388608
+```
+
+## See Also
+
+[hawq load](hawqload.html#topic1)
+
+
diff --git a/reference/cli/admin_utilities/hawqfilespace.html.md.erb b/reference/cli/admin_utilities/hawqfilespace.html.md.erb
new file mode 100644
index 0000000..97a461d
--- /dev/null
+++ b/reference/cli/admin_utilities/hawqfilespace.html.md.erb
@@ -0,0 +1,132 @@
+---
+title: hawq filespace
+---
+
+Creates a filespace using a configuration file that defines a file system location. Filespaces describe the physical file system resources to be used by a tablespace.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+hawq filespace [-c | --config fs_config_file ]
+hawq filespace [-o | --output output_directory_name ]
+hawq filespace [--movefilespace FILESPACE]
+hawq filespace [--location dfslocation]
+hawq filespace [-l | --logdir logfile_directory]
+hawq filespace [-h | --host hostname]
+hawq filespace [-p | --port PORT]
+hawq filespace [-U | --username NAME ]
+hawq filespace [-W | --password]
+
+hawq filespace -v | --version
+
+hawq filespace -? | --help
+```
+
+## Description<a id="topic1__section3"></a>
+
+A tablespace requires a file system location to store its database files. This file system location for all components in a HAWQ system is referred to as a *filespace*. Once a filespace is defined, it can be used by one or more tablespaces.
+
+The `--movefilespace` option allows you to relocate a filespace and its components within a dfs file system.
+
+When used with the `-o` option, the `hawq filespace` utility looks up your system configuration information in the system catalog tables and prompts you for the appropriate file system location needed to create the filespace. It then outputs a configuration file that can be used to create a filespace. If a file name is not specified, a `hawqfilespace_config_`*\#* file will be created in the current directory by default.
+
+Once you have a configuration file, you can run `hawq filespace` with the `-c` option to create the filespace in HAWQ system.
+
+**Note:** If segments are down due to a power or nic failure, you may see inconsistencies during filespace creation. You may not be able to bring up the cluster.
+
+## Options<a id="topic1__section4"></a>
+
+-c | --config *fs\_config\_file*
+A configuration file containing:
+
+- An initial line denoting the new filespace name. For example:
+
+ filespace:*myfs*
+
+-o | --output *output\_directory\_name*
+The directory location and file name to output the generated filespace configuration file. You will be prompted to enter a name for the filespace and file system location. The file system locations must exist on all hosts in your system prior to running the `hawq filespace` command. You will specify the number of replicas to create. The default is 3 replicas. After the utility creates the configuration file, you can manually edit the file to make any required changes to the filespace layout before creating the filespace in HAWQ.
+
+--movefilespace *FILESPACE*
+Create the filespace in a new location on a distributed file system. Updates the dfs url in the HAWQ database, so that data in the original location can be moved or deleted. Data in the original location is not affected by this command.
+
+--location *dfslocation*
+Specifies the new URL location to which a dfs file system should be moved.
+
+-l | --logdir *logfile\_directory*
+The directory to write the log file. Defaults to `~/hawqAdminLogs`.
+
+-v | --version (show utility version)
+Displays the version of this utility.
+
+-? | --help (help)
+Displays the command usage and syntax.
+
+**Connection Options**
+
+-h | --host *hostname*
+The host name of the machine on which the HAWQ master database server is running. If not specified, reads from the environment variable `PGHOST` or defaults to localhost.
+
+-p | --port *port*
+The TCP port on which the HAWQ master database server is listening for connections. If not specified, reads from the environment variable `PGPORT` or defaults to 5432.
+
+-U | --username *superuser\_name*
+The database superuser role name to connect as. If not specified, reads from the environment variable `PGUSER` or defaults to the current system user name. Only database superusers are allowed to create filespaces.
+
+-W | --password
+Force a password prompt.
+
+## Example 1<a id="topic1__section6"></a>
+
+Create a filespace configuration file. Depending on your system setup, you may need to specify the host and port. You will be prompted to enter a name for the filespace and a replica number. You will then be asked for the DFS location. The file system locations must exist on all hosts in your system prior to running the `hawq filespace` command:
+
+``` pre
+$ hawq filespace -o .
+Enter a name for this filespace
+> fastdisk
+Enter replica num for filespace. If 0, default replica num is used (default=3)
+0
+Please specify the DFS location for the filespace (for example: localhost:9000/fs)
+location> localhost:9000/hawqfs
+
+20160203:11:35:42:272716 hawqfilespace:localhost:gpadmin-[INFO]:-[created]
+20160203:11:35:42:272716 hawqfilespace:localhost:gpadmin-[INFO]:-
+To add this filespace to the database please run the command:
+ hawqfilespace --config ./hawqfilespace_config_20160203_112711
+Checking your configuration:
+
+Your system has 1 hosts with 2 primary segments
+per host.
+
+Configuring hosts: [sdw1, sdw2]
+
+Enter a file system location for the master:
+master location> /hawq_master_filespc
+```
+
+Example filespace configuration file:
+
+``` pre
+filespace:fastdisk
+mdw:1:/hawq_master_filespc/gp-1
+sdw1:2:/hawq_pri_filespc/gp0
+sdw2:3:/hawq_pri_filespc/gp1
+```
+
+Execute the configuration file to create the filespace:
+
+``` pre
+$ hawq filespace --config hawq_filespace_config_1
+```
+
+## Example 2
+
+Create the filespace at `cdbfast_fs_a` and move an hdfs filesystem to it:
+
+``` pre
+$ hawq filespace --movefilespace=cdbfast_fs_a
+ --location=hdfs://gphd-cluster/cdbfast_fs_a/
+```
+
+## See Also<a id="topic1__section7"></a>
+
+[CREATE TABLESPACE](../../sql/CREATE-TABLESPACE.html)
diff --git a/reference/cli/admin_utilities/hawqinit.html.md.erb b/reference/cli/admin_utilities/hawqinit.html.md.erb
new file mode 100644
index 0000000..989fa93
--- /dev/null
+++ b/reference/cli/admin_utilities/hawqinit.html.md.erb
@@ -0,0 +1,143 @@
+---
+title: hawq init
+---
+
+The `hawq init cluster` command initializes a HAWQ system and starts it.
+
+The master or segment nodes can be individually initialized by using `hawq init master` and `hawq init segment` commands, respectively. The `hawq init standby` command initializes a standby master host for a HAWQ system.
+
+The `hawq init object --standby-host=` option allows defining the host for a standby at initialization.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+hawq init object [--options]
+```
+
+``` pre
+hawq init object
+
+ [--locale=locale] [--lc-collate=locale]
+ [--lc-ctype=locale] [--lc-messages=locale]
+ [--lc-monetary=locale] [--lc-numeric=locale]
+ [--lc-time=locale]
+ [--bucket_number number]
+ [--max_connections=number]
+ [--shared_buffers number]
+ [standby | cluster --standby-host address_of_standby_host
+ [-a] [-l logfile_directory] [-q] [-v] [-t]
+[-n]
+hawq init -? --help
+```
+
+## Description<a id="topic1__section3"></a>
+
+The `hawq init <object>` utility will create a HAWQ instance using configuration parameters defined in `$GPHOME/etc/hawq-site.xml`. Before running this utility, make sure that you have installed the HAWQ software on all the hosts in the array.
+
+In a HAWQ DBMS, each database instance (the master and all segments) must be initialized across all of the hosts in the system in such a way that they can all work together as a unified DBMS. The `hawq init cluster` utility takes care of initializing the HAWQ master and each segment instance, and configuring the system as a whole. When hawq init cluster is run, the cluster comes online automatically without needing to explicitly start it. A single node cluster can be started without any user-defined changes to the default `hawq-site.xml` file. The template-hawq-site.xml file is used to specify the configuration for larger clusters.
+
+To use the template for initializing a new cluster configuration, replace the items contained within the % markers, for example in: `value%master.host%value`, `%master.host%` would be replaced with the master host name. After modification, rename the file to the name of the default configuration file: `hawq-site.xml`.
+
+Before running `hawq-site.xml`, you must set the `$GPHOME` environment variable to point to the location of your HAWQ installation on the master host and exchange SSH keys between all host addresses in the array using `hawq ssh-exkeys`.
+
+This utility performs the following tasks:
+
+- Verifies that the parameters in the configuration file are correct.
+- Ensures that a connection can be established to each host address. If a host address cannot be reached, the utility will exit.
+- Verifies the locale settings.
+- Initializes the master instance.
+- Initializes the standby master instance (if specified).
+- Initializes the segment instances.
+- Configures the HAWQ system and checks for errors.
+- Starts the HAWQ system.
+
+The `hawq init standby` utility can be run on the currently active *primary* master host or on the standby node.
+
+`hawq init standby` performs the following steps:
+
+- Updates the HAWQ system catalog to add the new standby master host information
+- Edits the `pg_hba.conf` file of the HAWQ master to allow access from the newly added standby master.
+- Sets up the standby master instance on the alternate master host
+- Starts the synchronization process
+
+A backup, standby master host serves as a 'warm standby' in the event of the primary master host becoming non-operational. The standby master is kept up to date by transaction log replication processes (the `walsender` and `walreceiver`), which run on the primary master and standby master hosts and keep the data between the primary and standby master hosts synchronized. To add a standby master to the system, use the command `hawq init standby`, for example `hawq init standby host09`. You can configure the standby hostname at initialization without needing to run hawq config by defining it using the --standby-host option. To create the standby above, you would specify `hawq init standby --standby-host=host09` or `hawq init cluster --standby-host=host09`.
+
+If the primary master fails, the log replication process is shut down, and the standby master can be activated in its place by using the `hawq activate standby` utility. Upon activation of the standby master, the replicated logs are used to reconstruct the state of the master host at the time of the last successfully committed transaction.
+
+## Objects
+
+cluster
+Start a HAWQ cluster.
+
+master
+Start HAWQ master.
+
+segment
+Start a local segment node.
+
+standby
+Start a HAWQ standby master.
+
+## Options<a id="topic1__section4"></a>
+
+-a (do not prompt)
+Do not prompt the user for confirmation.
+
+-l --logdir*logfile\_directory*
+The directory to write the log file. Defaults to `~/hawq/AdminLogs`.
+
+-q --quiet (no screen output)
+Run in quiet mode. Command output is not displayed on the screen, but is still written to the log file.
+
+-v --verbose
+Displays detailed status, progress and error messages and writes them to the log files.
+
+-t --timeout
+Sets timeout value in seconds. The default is 60 seconds.
+
+-n --no-update
+Resync the standby with the master, but do not update system catalog tables.
+
+--locale=*locale*
+Sets the default locale used by HAWQ. If not specified, the `LC_ALL`, `LC_COLLATE`, or `LANG` environment variable of the master host determines the locale. If these are not set, the default locale is `C` (`POSIX`). A locale identifier consists of a language identifier and a region identifier, and optionally a character set encoding. For example, `sv_SE` is Swedish as spoken in Sweden, `en_US` is U.S. English, and `fr_CA` is French Canadian. If more than one character set can be useful for a locale, then the specifications look like this: `en_US.UTF-8` (locale specification and character set encoding). On most systems, the command `locale` will show the locale environment settings and `locale -a` will show a list of all available locales.
+
+--lc-collate=*locale*
+Similar to `--locale`, but sets the locale used for collation (sorting data). The sort order cannot be changed after HAWQ is initialized, so it is important to choose a collation locale that is compatible with the character set encodings that you plan to use for your data. There is a special collation name of `C` or `POSIX` (byte-order sorting as opposed to dictionary-order sorting). The `C` collation can be used with any character encoding.
+
+--lc-ctype=*locale*
+Similar to `--locale`, but sets the locale used for character classification (what character sequences are valid and how they are interpreted). This cannot be changed after HAWQ is initialized, so it is important to choose a character classification locale that is compatible with the data you plan to store in HAWQ.
+
+--lc-messages=*locale*
+Similar to `--locale`, but sets the locale used for messages output by HAWQ. The current version of HAWQ does not support multiple locales for output messages (all messages are in English), so changing this setting will not have any effect.
+
+--lc-monetary=*locale*
+Similar to `--locale`, but sets the locale used for formatting currency amounts.
+
+--lc-numeric=*locale*
+Similar to `--locale`, but sets the locale used for formatting numbers.
+
+--lc-time=*locale*
+Similar to `--locale`, but sets the locale used for formatting dates and times.
+
+--bucket\_number=*number*
+Sets value of `default_hash_table_bucket_number`, which sets the default number of hash buckets for creating virtual segments. This parameter overrides the default value of `default_hash_table_bucket_number` set in `hawq-site.xml` by an Ambari install. If not specified, `hawq init` will use the value in `hawq-site.xml`.
+
+--max\_connections=*number*
+Sets the number of client connections allowed to the master. The default is 250.
+
+--shared\_buffers *number*
+Sets the number of shared\_buffers to be used when initializing HAWQ.
+
+-s --standby-host *name\_of\_standby\_host*
+Adds a standby host name to hawq-site.xml and syncs it to all the nodes. If a standby host name was already defined in hawq-site.xml, using this option will overwrite the existing value.
+
+-? --help
+Displays the online help.
+
+## Examples<a id="topic1__section6"></a>
+
+Initialize a HAWQ array with an optional standby master host:
+
+``` pre
+$ hawq init standby
+```
diff --git a/reference/cli/admin_utilities/hawqload.html.md.erb b/reference/cli/admin_utilities/hawqload.html.md.erb
new file mode 100644
index 0000000..4e43621
--- /dev/null
+++ b/reference/cli/admin_utilities/hawqload.html.md.erb
@@ -0,0 +1,401 @@
+---
+title: hawq load
+---
+
+Acts as an interface to the external table parallel loading feature. Executes a load specification defined in a YAML-formatted control file to invoke the HAWQ parallel file server (`gpfdist`).
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+hawq load -f control_file [-l log_file] [-h hostname] [-p port]
+ [-U username] [-d database] [-W]
+ [--gpfdist_timeout seconds] [[-v | -V] [-q]] [-D]
+
+hawq load -?
+
+hawq load --version
+```
+
+## Prerequisites<a id="topic1__section3"></a>
+
+The client machine where `hawq load` is executed must have the following:
+
+- Python 2.6.2 or later, `pygresql` (the Python interface to PostgreSQL), and `pyyaml`. Note that Python and the required Python libraries are included with the HAWQ server installation, so if you have HAWQ installed on the machine where `hawq load` is running, you do not need a separate Python installation.
+ **Note:** HAWQ Loaders for Windows supports only Python 2.5 (available from [www.python.org](http://python.org)).
+
+- The [gpfdist](gpfdist.html#topic1) parallel file distribution program installed and in your `$PATH`. This program is located in `$GPHOME/bin` of your HAWQ server installation.
+- Network access to and from all hosts in your HAWQ array (master and segments).
+- Network access to and from the hosts where the data to be loaded resides (ETL servers).
+
+## Description<a id="topic1__section4"></a>
+
+`hawq load` is a data loading utility that acts as an interface to HAWQ's external table parallel loading feature. Using a load specification defined in a YAML formatted control file, `hawq load` executes a load by invoking the HAWQ parallel file server ([gpfdist](gpfdist.html#topic1)), creating an external table definition based on the source data defined, and executing an `INSERT`, `UPDATE` or `MERGE` operation to load the source data into the target table in the database.
+
+The operation, including any SQL commands specified in the `SQL` collection of the YAML control file (see [Control File Format](#topic1__section7)), are performed as a single transaction to prevent inconsistent data when performing multiple, simultaneous load operations on a target table.
+
+## Options<a id="topic1__section5"></a>
+
+-f *control\_file*
+**Required.** A YAML file that contains the load specification details. See [Control File Format](#topic1__section7).
+
+--gpfdist\_timeout *seconds*
+Sets the timeout for the `gpfdist` parallel file distribution program to send a response. Enter a value from `0` to `30` seconds (entering "`0`" to disables timeouts). Note that you might need to increase this value when operating on high-traffic networks.
+
+-l *log\_file*
+Specifies where to write the log file. Defaults to `~/hawq/Adminlogs/hawq_load_YYYYMMDD`. For more information about the log file, see [Log File Format](#topic1__section9).
+
+-q (no screen output)
+Run in quiet mode. Command output is not displayed on the screen, but is still written to the log file.
+
+-D (debug mode)
+Check for error conditions, but do not execute the load.
+
+-v (verbose mode)
+Show verbose output of the load steps as they are executed.
+
+-V (very verbose mode)
+Shows very verbose output.
+
+-? (show help)
+Show help, then exit.
+
+--version
+Show the version of this utility, then exit.
+
+**Connection Options**
+
+-d *database*
+The database to load into. If not specified, reads from the load control file, the environment variable `$PGDATABASE` or defaults to the current system user name.
+
+-h *hostname*
+Specifies the host name of the machine on which the HAWQ master database server is running. If not specified, reads from the load control file, the environment variable `$PGHOST` or defaults to `localhost`.
+
+-p *port*
+Specifies the TCP port on which the HAWQ master database server is listening for connections. If not specified, reads from the load control file, the environment variable `$PGPORT` or defaults to 5432.
+
+-U *username*
+The database role name to connect as. If not specified, reads from the load control file, the environment variable `$PGUSER` or defaults to the current system user name.
+
+-W (force password prompt)
+Force a password prompt. If not specified, reads the password from the environment variable `$PGPASSWORD` or from a password file specified by `$PGPASSFILE` or in `~/.pgpass`. If these are not set, then `hawq load` will prompt for a password even if `-W` is not supplied.
+
+## Control File Format<a id="topic1__section7"></a>
+
+The `hawq load` control file uses the [YAML 1.1](http://yaml.org/spec/1.1/) document format and then implements its own schema for defining the various steps of a HAWQ load operation. The control file must be a valid YAML document.
+
+The `hawq load` program processes the control file document in order and uses indentation (spaces) to determine the document hierarchy and the relationships of the sections to one another. The use of white space is significant. White space should not be used simply for formatting purposes, and tabs should not be used at all.
+
+The basic structure of a load control file is:
+
+``` pre
+---
+VERSION: 1.0.0.1
+DATABASE: db_name
+USER: db_username
+HOST: master_hostname
+PORT: master_port
+GPLOAD:
+ INPUT:
+ - SOURCE:
+ LOCAL_HOSTNAME:
+ - hostname_or_ip
+ PORT: http_port
+ | PORT_RANGE: [start_port_range, end_port_range]
+ FILE:
+ - /path/to/input_file
+ SSL: true | false
+ CERTIFICATES_PATH: /path/to/certificates
+ - COLUMNS:
+ - field_name: data_type
+ - TRANSFORM: 'transformation'
+ - TRANSFORM_CONFIG: 'configuration-file-path'
+ - MAX_LINE_LENGTH: integer
+ - FORMAT: text | csv
+ - DELIMITER: 'delimiter_character'
+ - ESCAPE: 'escape_character' | 'OFF'
+ - NULL_AS: 'null_string'
+ - FORCE_NOT_NULL: true | false
+ - QUOTE: 'csv_quote_character'
+ - HEADER: true | false
+ - ENCODING: database_encoding
+ - ERROR_LIMIT: integer
+ - ERROR_TABLE: schema.table_name
+ OUTPUT:
+ - TABLE: schema.table_name
+ - MODE: insert | update | merge
+ - MATCH_COLUMNS:
+ - target_column_name
+ - UPDATE_COLUMNS:
+ - target_column_name
+ - UPDATE_CONDITION: 'boolean_condition'
+ - MAPPING:
+ target_column_name: source_column_name | 'expression'
+ PRELOAD:
+ - TRUNCATE: true | false
+ - REUSE_TABLES: true | false
+ SQL:
+ - BEFORE: "sql_command"
+ - AFTER: "sql_command"
+```
+
+**Control File Schema Elements**
+
+The control file contains the schema elements for:
+
+- Version
+- Database
+- User
+- Host
+- Port
+- GPLOAD file
+
+**VERSION**
+Optional. The version of the `hawq load` control file schema, for example: 1.0.0.1.
+
+**DATABASE**
+Optional. Specifies which database in HAWQ to connect to. If not specified, defaults to `$PGDATABASE` if set or the current system user name. You can also specify the database on the command line using the `-d` option.
+
+**USER**
+Optional. Specifies which database role to use to connect. If not specified, defaults to the current user or `$PGUSER` if set. You can also specify the database role on the command line using the `-U` option.
+
+If the user running `hawq load` is not a HAWQ superuser, then the server configuration parameter `gp_external_grant_privileges` must be set to `on` for the load to be processed.
+
+**HOST**
+Optional. Specifies HAWQ master host name. If not specified, defaults to localhost or `$PGHOST` if set. You can also specify the master host name on the command line using the `-h` option.
+
+**PORT**
+Optional. Specifies HAWQ master port. If not specified, defaults to 5432 or `$PGPORT` if set. You can also specify the master port on the command line using the `-p` option.
+
+**GPLOAD**
+Required. Begins the load specification section. A `GPLOAD` specification must have an `INPUT` and an `OUTPUT` section defined.
+
+INPUT -
+Required. Defines the location and the format of the input data to be loaded. `hawq load` will start one or more instances of the [gpfdist](gpfdist.html#topic1) file distribution program on the current host and create the required external table definition(s) in HAWQ that point to the source data. Note that the host from which you run `hawq load` must be accessible over the network by all HAWQ hosts (master and segments).
+
+SOURCE
+Required. The `SOURCE` block of an `INPUT` specification defines the location of a source file. An `INPUT` section can have more than one `SOURCE` block defined. Each `SOURCE` block defined corresponds to one instance of the [gpfdist](gpfdist.html#topic1) file distribution program that will be started on the local machine. Each `SOURCE` block defined must have a `FILE` specification.
+
+LOCAL\_HOSTNAME
+Optional. Specifies the host name or IP address of the local machine on which `hawq load` is running. If this machine is configured with multiple network interface cards (NICs), you can specify the host name or IP of each individual NIC to allow network traffic to use all NICs simultaneously. The default is to use the local machine's primary host name or IP only.
+
+PORT
+Optional. Specifies the specific port number that the [gpfdist](gpfdist.html#topic1) file distribution program should use. You can also supply a `PORT_RANGE` to select an available port from the specified range. If both `PORT` and `PORT_RANGE` are defined, then `PORT` takes precedence. If neither `PORT` or `PORT_RANGE` are defined, the default is to select an available port between 8000 and 9000.
+
+If multiple host names are declared in `LOCAL_HOSTNAME`, this port number is used for all hosts. This configuration is desired if you want to use all NICs to load the same file or set of files in a given directory location.
+
+PORT\_RANGE
+Optional. Can be used instead of `PORT` to supply a range of port numbers from which `hawq load` can choose an available port for this instance of the [gpfdist](gpfdist.html#topic1) file distribution program.
+
+FILE
+Required. Specifies the location of a file, named pipe, or directory location on the local file system that contains data to be loaded. You can declare more than one file so long as the data is of the same format in all files specified.
+
+If the files are compressed using `gzip` or `bzip2` (have a `.gz` or `.bz2` file extension), the files will be uncompressed automatically (provided that `gunzip` or `bunzip2` is in your path).
+
+When specifying which source files to load, you can use the wildcard character (`*`) or other C-style pattern matching to denote multiple files. The files specified are assumed to be relative to the current directory from which `hawq load` is executed (or you can declare an absolute path).
+
+SSL
+Optional. Specifies usage of SSL encryption.
+
+CERTIFICATES\_PATH
+Required when SSL is `true`; cannot be specified when SSL is `false` or unspecified. The location specified in `CERTIFICATES_PATH` must contain the following files:
+
+- The server certificate file, `server.crt`
+- The server private key file, `server.key`
+- The trusted certificate authorities, `root.crt`
+
+The root directory (`/`) cannot be specified as `CERTIFICATES_PATH`.
+
+COLUMNS
+Optional. Specifies the schema of the source data file(s) in the format of `field_name:data_type`. The `DELIMITER` character in the source file is what separates two data value fields (columns). A row is determined by a line feed character (`0x0a`).
+
+If the input `COLUMNS` are not specified, then the schema of the output `TABLE` is implied, meaning that the source data must have the same column order, number of columns, and data format as the target table.
+
+The default source-to-target mapping is based on a match of column names as defined in this section and the column names in the target `TABLE`. This default mapping can be overridden using the `MAPPING` section.
+
+TRANSFORM
+Optional. Specifies the name of the input XML transformation passed to `hawq load`. <span class="ph">For more information about XML transformations, see ["Loading and Unloading Data."](../../../datamgmt/load/g-loading-and-unloading-data.html#topic1).</span>
+
+TRANSFORM\_CONFIG
+Optional. Specifies the location of the XML transformation configuration file that is specified in the `TRANSFORM` parameter, above.
+
+MAX\_LINE\_LENGTH
+Optional. An integer that specifies the maximum length of a line in the XML transformation data passed to `hawq load`.
+
+FORMAT
+Optional. Specifies the format of the source data file(s) - either plain text (`TEXT`) or comma separated values (`CSV`) format. Defaults to `TEXT` if not specified.<span class="ph"> For more information about the format of the source data, see ["Loading and Unloading Data"](../../../datamgmt/load/g-loading-and-unloading-data.html#topic1) .</span>
+
+DELIMITER
+Optional. Specifies a single ASCII character that separates columns within each row (line) of data. The default is a tab character in TEXT mode, a comma in CSV mode.You can also specify a non-printable ASCII character via an escape sequence\\ using the decimal representation of the ASCII character. For example, `\014` represents the shift out character..
+
+ESCAPE
+Specifies the single character that is used for C escape sequences (such as `\n`, `\t`, `\100`, and so on) and for escaping data characters that might otherwise be taken as row or column delimiters. Make sure to choose an escape character that is not used anywhere in your actual column data. The default escape character is a \\ (backslash) for text-formatted files and a `"` (double quote) for csv-formatted files, however it is possible to specify another character to represent an escape. It is also possible to disable escaping in text-formatted files by specifying the value `'OFF'` as the escape value. This is very useful for data such as text-formatted web log data that has many embedded backslashes that are not intended to be escapes.
+
+NULL\_AS
+Optional. Specifies the string that represents a null value. The default is `\N` (backslash-N) in `TEXT` mode, and an empty value with no quotations in `CSV` mode. You might prefer an empty string even in `TEXT` mode for cases where you do not want to distinguish nulls from empty strings. Any source data item that matches this string will be considered a null value.
+
+FORCE\_NOT\_NULL
+Optional. In CSV mode, processes each specified column as though it were quoted and hence not a NULL value. For the default null string in CSV mode (nothing between two delimiters), this causes missing values to be evaluated as zero-length strings.
+
+QUOTE
+Required when `FORMAT` is `CSV`. Specifies the quotation character for `CSV` mode. The default is double-quote (`"`).
+
+HEADER
+Optional. Specifies that the first line in the data file(s) is a header row (contains the names of the columns) and should not be included as data to be loaded. If using multiple data source files, all files must have a header row. The default is to assume that the input files do not have a header row.
+
+ENCODING
+Optional. Character set encoding of the source data. Specify a string constant (such as `'SQL_ASCII'`), an integer encoding number, or `'DEFAULT'` to use the default client encoding. If not specified, the default client encoding is used.
+
+ERROR\_LIMIT
+Optional. Sets the error limit count for HAWQ segment instances during input processing. Error rows will be written to the table specified in `ERROR_TABLE`. The value of ERROR\_LIMIT must be 2 or greater.
+
+ERROR\_TABLE
+Optional when `ERROR_LIMIT` is declared. Specifies an error table where rows with formatting errors will be logged when running in single row error isolation mode. You can then examine this error table to see error rows that were not loaded (if any). If the `ERROR_TABLE` specified already exists, it will be used. If it does not exist, it will be automatically generated.
+
+For more information about handling load errors, see "[Loading and Unloading Data](../../../datamgmt/load/g-loading-and-unloading-data.html#topic1)".
+
+OUTPUT -
+Required. Defines the target table and final data column values that are to be loaded into the database.
+
+TABLE
+Required. The name of the target table to load into.
+
+MODE
+Optional. Defaults to `INSERT` if not specified. There are three available load modes:
+
+<span class="keyword option">INSERT</span> - Loads data into the target table using the following method:
+
+``` pre
+INSERT INTO target_table SELECT * FROM input_data;
+```
+
+<span class="keyword option">UPDATE</span> - Updates the `UPDATE_COLUMNS` of the target table where the rows have `MATCH_COLUMNS` attribute values equal to those of the input data, and the optional `UPDATE_CONDITION` is true.
+
+<span class="keyword option">MERGE</span> - Inserts new rows and updates the `UPDATE_COLUMNS` of existing rows where `MATCH_COLUMNS` attribute values are equal to those of the input data, and the optional `UPDATE_CONDITION` is true. New rows are identified when the `MATCH_COLUMNS` value in the source data does not have a corresponding value in the existing data of the target table. In those cases, the **entire row** from the source file is inserted, not only the `MATCH` and `UPDATE` columns. If there are multiple new `MATCH_COLUMNS` values that are the same, only one new row for that value will be inserted. Use `UPDATE_CONDITION` to filter out the rows to discard.
+
+MATCH\_COLUMNS
+Required if `MODE` is `UPDATE` or `MERGE`. Specifies the column(s) to use as the join condition for the update. The attribute value in the specified target column(s) must be equal to that of the corresponding source data column(s) in order for the row to be updated in the target table.
+
+UPDATE\_COLUMNS
+Required if `MODE` is `UPDATE` or `MERGE`. Specifies the column(s) to update for the rows that meet the `MATCH_COLUMNS` criteria and the optional `UPDATE_CONDITION`.
+
+UPDATE\_CONDITION
+Optional. Specifies a Boolean condition (similar to what you would declare in a `WHERE` clause) that must be met for a row in the target table to be updated (or inserted in the case of a `MERGE`).
+
+MAPPING
+Optional. If a mapping is specified, it overrides the default source-to-target column mapping. The default source-to-target mapping is based on a match of column names as defined in the source `COLUMNS` section and the column names of the target `TABLE`. A mapping is specified as either:
+
+`target_column_name: source_column_name`
+
+or
+
+`target_column_name: 'expression'`
+
+Where *expression* is any expression that you would specify in the `SELECT` list of a query, such as a constant value, a column reference, an operator invocation, a function call, and so on.
+
+PRELOAD
+Optional. Specifies operations to run prior to the load operation. Currently, the only preload operation is `TRUNCATE`.
+
+TRUNCATE
+Optional. If set to true, `hawq load` will remove all rows in the target table prior to loading it.
+
+REUSE\_TABLES
+Optional. If set to true, `hawq load` will not drop the external table objects and staging table objects it creates. These objects will be reused for future load operations that use the same load specifications. Reusing objects improves performance of trickle loads (ongoing small loads to the same target table).
+
+SQL
+Optional. Defines SQL commands to run before and/or after the load operation. Commands that contain spaces or special characters must be enclosed in quotes. You can specify multiple `BEFORE` and/or `AFTER` commands. List commands in the desired order of execution.
+
+BEFORE
+Optional. A SQL command to run before the load operation starts. Enclose commands in quotes.
+
+AFTER
+Optional. A SQL command to run after the load operation completes. Enclose commands in quotes.
+
+## Notes
+
+If your database object names were created using a double-quoted identifier (delimited identifier), you must specify the delimited name within single quotes in the `hawq load` control file. For example, if you create a table as follows:
+
+``` pre
+CREATE TABLE "MyTable" ("MyColumn" text);
+```
+
+Your YAML-formatted `hawq load` control file would refer to the above table and column names as follows:
+
+``` pre
+- COLUMNS:
+ - '"MyColumn"': text
+OUTPUT:
+ - TABLE: public.'"MyTable"'
+```
+
+## Log File Format<a id="topic1__section9"></a>
+
+Log files output by `hawq load` have the following format:
+
+``` pre
+timestamp|level|message
+```
+
+Where *timestamp* takes the form: `YYYY-MM-DD HH:MM:SS`, *level* is one of `DEBUG`, `LOG`, `INFO`, `ERROR`, and *message* is a normal text message.
+
+Some `INFO` messages that may be of interest in the log files are (where *\#* corresponds to the actual number of seconds, units of data, or failed rows):
+
+``` pre
+INFO|running time: #.## seconds
+INFO|transferred #.# kB of #.# kB.
+INFO|hawq load succeeded
+INFO|hawq load succeeded with warnings
+INFO|hawq load failed
+INFO|1 bad row
+INFO|# bad rows
+```
+
+## Examples<a id="topic1__section10"></a>
+
+Run a load job as defined in `my_load.yml`:
+
+``` pre
+hawq load -f my_load.yml
+```
+
+Example load control file:
+
+``` pre
+---
+VERSION: 1.0.0.1
+DATABASE: ops
+USER: gpadmin
+HOST: mdw-1
+PORT: 5432
+GPLOAD:
+ INPUT:
+ - SOURCE:
+ LOCAL_HOSTNAME:
+ - etl1-1
+ - etl1-2
+ - etl1-3
+ - etl1-4
+ PORT: 8081
+ FILE:
+ - /var/load/data/*
+ - COLUMNS:
+ - name: text
+ - amount: float4
+ - category: text
+ - desc: text
+ - date: date
+ - FORMAT: text
+ - DELIMITER: '|'
+ - ERROR_LIMIT: 25
+ - ERROR_TABLE: payables.err_expenses
+ OUTPUT:
+ - TABLE: payables.expenses
+ - MODE: INSERT
+ SQL:
+ - BEFORE: "INSERT INTO audit VALUES('start', current_timestamp)"
+ - AFTER: "INSERT INTO audit VALUES('end',
+current_timestamp)"
+```
+
+## See Also<a id="topic1__section11"></a>
+
+[gpfdist](gpfdist.html#topic1), [CREATE EXTERNAL TABLE](../../sql/CREATE-EXTERNAL-TABLE.html#topic1)
diff --git a/reference/cli/admin_utilities/hawqrestart.html.md.erb b/reference/cli/admin_utilities/hawqrestart.html.md.erb
new file mode 100644
index 0000000..41ebda3
--- /dev/null
+++ b/reference/cli/admin_utilities/hawqrestart.html.md.erb
@@ -0,0 +1,112 @@
+---
+title: hawq restart
+---
+
+Shuts down and then restarts a HAWQ system after shutdown is complete.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+hawq restart object [-l|--logdir] [-q|--quiet] [-v|--verbose]
+ [-M|--mode smart | fast | immediate] [-u|--reload] [-m|--masteronly] [-R|--restrict]
+ [-t|--timeout timeout_seconds] [-U | --special-mode maintenance]
+ [--ignore_bad-hosts] cluster | allsegments
+
+```
+
+``` pre
+hawq restart -? | -h | --help
+
+hawq restart --version
+```
+
+## Description<a id="topic1__section3"></a>
+
+The `hawq restart` utility is used to shut down and restart the HAWQ server processes. It is essentially equivalent to performing a `hawq stop -M smart` operation followed by `hawq start`.
+
+The *object* in the command specifies what entity should be started: e.g. a cluster, a segment, the master node, standby node, or all segments in the cluster.
+
+When the `hawq restart` command runs, the utility uploads changes made to the master `pg_hba.conf` file or to *runtime* configuration parameters in the master `hawq-site.xml` file without interruption of service. Note that any active sessions will not pick up the changes until they reconnect to the database.
+
+## Objects
+
+cluster
+Restart a HAWQ cluster.
+
+master
+Restart HAWQ master.
+
+segment
+Restart a local segment node.
+
+standby
+Restart a HAWQ standby.
+
+allsegments
+Restart all segments.
+
+## Options<a id="topic1__section4"></a>
+
+-a (do not prompt)
+Do not prompt the user for confirmation.
+
+-l --logdir *logfile\_directory*
+Specifies the log directory for logs of the management tools. The default is `~/hawq/Adminlogs/`.
+
+-q --quiet
+Run in quiet mode. Command output is not displayed on the screen, but is still written to the log file.
+
+-v --verbose
+Displays detailed status, progress and error messages output by the utility.
+
+-t --timeout *timeout\_seconds*
+Specifies a timeout in seconds to wait for a segment instance to start up. If a segment instance was shutdown abnormally (due to power failure or killing its `postgres` database listener process, for example), it may take longer to start up due to the database recovery and validation process. If not specified, the default timeout is 60 seconds.
+
+-M --mode smart | fast | immediate
+Smart shutdown is the default. Shutdown fails with a warning message, if active connections are found.
+
+Fast shut down interrupts and rolls back any transactions currently in progress .
+
+Immediate shutdown aborts transactions in progress and kills all `postgres` processes without allowing the database server to complete transaction processing or clean up any temporary or in-process work files. Because of this, immediate shutdown is not recommended. In some instances, it can cause database corruption that requires manual recovery.
+
+-u --reload
+Utility mode. This mode runs on the master, only, and only allows incoming sessions that specify gp\_session\_role=utility. It allows bash scripts to reload the parameter values and connect but protects the system from normal clients who might be trying to connect to the system during startup.
+
+-R --restrict
+Starts HAWQ in restricted mode (only database superusers are allowed to connect).
+
+-U --special-mode maintenance
+(Superuser only) Start HAWQ in \[maintenance | upgrade\] mode. In maintenance mode, the `gp_maintenance_conn` parameter is set.
+
+--ignore\_bad\_hosts cluster | allsegments
+Overrides copying configuration files to a host on which SSH validation fails. If ssh to a skipped host is reestablished, make sure the configuration files are re-synched once it is reachable.
+
+-? | -h | --help (help)
+Displays the online help.
+
+--version (show utility version)
+Displays the version of this utility.
+
+## Examples<a id="topic1__section5"></a>
+
+Restart a HAWQ cluster:
+
+``` pre
+hawq restart cluster
+```
+
+Restart a HAWQ system in restricted mode (only allow superuser connections):
+
+``` pre
+hawq restart cluster -R
+```
+
+Start the HAWQ master instance only and connect in utility mode:
+
+``` pre
+hawq start master -m PGOPTIONS='-c gp_session_role=utility' psql
+```
+
+## See Also<a id="topic1__section6"></a>
+
+[hawq stop](hawqstop.html#topic1), [hawq start](hawqstart.html#topic1)
diff --git a/reference/cli/admin_utilities/hawqscp.html.md.erb b/reference/cli/admin_utilities/hawqscp.html.md.erb
new file mode 100644
index 0000000..9fbf637
--- /dev/null
+++ b/reference/cli/admin_utilities/hawqscp.html.md.erb
@@ -0,0 +1,81 @@
+---
+title: hawq scp
+---
+
+Copies files between multiple hosts at once.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+hawq scp { -f hostfile_hawqssh | -h hostname [-h hostname ...] }
+ [-J character] [-v] [[user@]hostname:]file_to_copy [...]
+ [[user@]hostname:]copy_to_path
+
+hawq scp -?
+
+hawq scp --version
+```
+
+## Description<a id="topic1__section3"></a>
+
+The `hawq scp` utility allows you to copy one or more files from the specified hosts to other specified hosts in one command using SCP (secure copy). For example, you can copy a file from the HAWQ master host to all of the segment hosts at the same time.
+
+To specify the hosts involved in the SCP session, use the `-f` option to specify a file containing a list of host names, or use the `-h` option to name single host names on the command-line. At least one host name (`-h`) or a host file (`-f`) is required. The `-J` option allows you to specify a single character to substitute for the *hostname* in the `copy from` and `copy to` destination strings. If `-J` is not specified, the default substitution character is an equal sign (`=`). For example, the following command will copy `.bashrc` from the local host to `/home/gpadmin` on all hosts named in `hostfile_gpssh`:
+
+``` pre
+hawq scp -f hostfile_hawqssh .bashrc =:/home/gpadmin
+```
+
+If a user name is not specified in the host list or with *user*`@` in the file path, `hawq scp` will copy files as the currently logged in user. To determine the currently logged in user, do a `whoami` command. By default, `hawq scp` goes to `$HOME` of the session user on the remote hosts after login. To ensure the file is copied to the correct location on the remote hosts, i use absolute paths.
+
+Before using `hawq scp`, you must have a trusted host setup between the hosts involved in the SCP session. You can use the utility `hawq ssh-exkeys` to update the known host files and exchange public keys between hosts if you have not done so already.
+
+## Options<a id="topic1__section4"></a>
+
+-f *hostfile\_hawqssh*
+Specifies the name of a file that contains a list of hosts that will participate in this SCP session. The syntax of the host file is one host per line as follows:
+
+``` pre
+<hostname>
+```
+
+-h *hostname*
+Specifies a single host name that will participate in this SCP session. You can use the `-h` option multiple times to specify multiple host names.
+
+-J *character*
+The `-J` option allows you to specify a single character to substitute for the *hostname* in the `copy from` and `copy to` destination strings. If `-J` is not specified, the default substitution character is an equal sign (`=`).
+
+-v (verbose mode)
+Optional. Reports additional messages in addition to the SCP command output.
+
+*file\_to\_copy*
+Required. The file name (or absolute path) of a file that you want to copy to other hosts (or file locations). This can be either a file on the local host or on another named host.
+
+*copy\_to\_path*
+Required. The path where you want the file(s) to be copied on the named hosts. If an absolute path is not used, the file will be copied relative to `$HOME` of the session user. You can also use the equal sign '`=`' (or another character that you specify with the `-J` option) in place of a *hostname*. This will then substitute in each host name as specified in the supplied host file (`-f`) or with the `-h` option.
+
+-? (help)
+Displays the online help.
+
+--version
+Displays the version of this utility.
+
+## Examples<a id="topic1__section5"></a>
+
+Copy the file named `installer.tar` to `/` on all the hosts in the file `hostfile_hawqssh`.
+
+``` pre
+hawq scp -f hostfile_hawqssh installer.tar =:/
+```
+
+Copy the file named *myfuncs.so* to the specified location on the hosts named `sdw1` and `sdw2`:
+
+``` pre
+hawq scp -h sdw1 -h sdw2 myfuncs.so =:/usr/local/-db/lib
+```
+
+## See Also
+
+[hawq ssh](hawqssh.html#topic1), [hawq ssh-exkeys](hawqssh-exkeys.html#topic1)
+
+
diff --git a/reference/cli/admin_utilities/hawqssh-exkeys.html.md.erb b/reference/cli/admin_utilities/hawqssh-exkeys.html.md.erb
new file mode 100644
index 0000000..a80a319
--- /dev/null
+++ b/reference/cli/admin_utilities/hawqssh-exkeys.html.md.erb
@@ -0,0 +1,102 @@
+---
+title: hawq ssh-exkeys
+---
+
+Exchanges SSH public keys between hosts.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+hawq ssh-exkeys -f hostfile_exkeys | - h hostname [-h hostname ...]
+
+hawq ssh-exkeys -e hostfile_exkeys -x hostfile_hawqexpand
+
+hawq ssh-exkeys -?
+
+hawq ssh-exkeys --version
+```
+
+## Description<a id="topic1__section3"></a>
+
+The `hawq ssh-exkeys` utility exchanges SSH keys between the specified host names (or host addresses). This allows SSH connections between HAWQ hosts and network interfaces without a password prompt. The utility is used to initially prepare a HAWQ system for password-free SSH access, and also to add additional ssh keys when expanding a HAWQ system.
+
+To specify the hosts involved in an initial SSH key exchange, use the `-f` option to specify a file containing a list of host names (recommended), or use the `-h` option to name single host names on the command-line. At least one host name (`-h`) or a host file is required. Note that the local host is included in the key exchange by default.
+
+To specify new expansion hosts to be added to an existing HAWQ system, use the `-e` and `-x` options. The `-e` option specifies a file containing a list of existing hosts in the system that already have SSH keys. The `-x` option specifies a file containing a list of new hosts that need to participate in the SSH key exchange.
+
+Keys are exchanged as the currently logged in user. A good practice is performing the key exchange process twice: once as `root` and once as the `gpadmin` user (the designated owner of your HAWQ /> installation). The HAWQ management utilities require that the same non-root user be created on all hosts in the HAWQ system, and the utilities must be able to connect as that user to all hosts without a password prompt.
+
+The `hawq ssh-exkeys` utility performs key exchange using the following steps:
+
+- Creates an RSA identification key pair for the current user if one does not already exist. The public key of this pair is added to the `authorized_keys` file of the current user.
+- Updates the `known_hosts` file of the current user with the host key of each host specified using the `-h`, `-f`, `-e`, and `-x` options.
+- Connects to each host using `ssh` and obtains the `authorized_keys`, `known_hosts`, and `id_rsa.pub` files to set up password-free access.
+- Adds keys from the `id_rsa.pub` files obtained from each host to the `authorized_keys` file of the current user.
+- Updates the `authorized_keys`, `known_hosts`, and `id_rsa.pub` files on all hosts with new host information (if any).
+
+## Options<a id="topic1__section4"></a>
+
+-e *hostfile\_exkeys*
+When doing a system expansion, this is the name and location of a file containing all configured host names and host addresses (interface names) for each host in your *current* HAWQ system (master, standby master and segments), one name per line without blank lines or extra spaces. Hosts specified in this file cannot be specified in the host file used with `-x`.
+
+-f *hostfile\_exkeys*
+Specifies the name and location of a file containing all configured host names and host addresses (interface names) for each host in your HAWQ system (master, standby master and segments), one name per line without blank lines or extra spaces.
+
+-h *hostname*
+Specifies a single host name (or host address) that will participate in the SSH key exchange. You can use the `-h` option multiple times to specify multiple host names and host addresses.
+
+--version
+Displays the version of this utility.
+
+-x *hostfile\_hawqexpand*
+When doing a system expansion, this is the name and location of a file containing all configured host names and host addresses (interface names) for each new segment host you are adding to your HAWQ system, one name per line without blank lines or extra spaces. Hosts specified in this file cannot be specified in the host file used with `-e`.
+
+-? (help)
+Displays the online help.
+
+## Examples<a id="topic1__section5"></a>
+
+Exchange SSH keys between all host names and addresses listed in the file `hostfile_exkeys`:
+
+``` pre
+$ hawq ssh-exkeys -f hostfile_exkeys
+```
+
+Exchange SSH keys between the hosts `sdw1`, `sdw2`, and `sdw3`:
+
+``` pre
+$ hawq ssh-exkeys -h sdw1 -h sdw2 -h sdw3
+```
+
+Exchange SSH keys between existing hosts `sdw1`, `sdw2`, and `sdw3`, and new hosts `sdw4` and `sdw5` as part of a system expansion operation:
+
+``` pre
+$ cat hostfile_exkeys
+mdw
+mdw-1
+mdw-2
+smdw
+smdw-1
+smdw-2
+sdw1
+sdw1-1
+sdw1-2
+sdw2
+sdw2-1
+sdw2-2
+sdw3
+sdw3-1
+sdw3-2
+$ cat hostfile_hawqexpand
+sdw4
+sdw4-1
+sdw4-2
+sdw5
+sdw5-1
+sdw5-2
+$ hawq ssh-exkeys -e hostfile_exkeys -x hostfile_hawqexpand
+```
+
+## See Also<a id="topic1__section6"></a>
+
+[hawq ssh](hawqssh.html#topic1), [hawq scp](hawqscp.html#topic1)
diff --git a/reference/cli/admin_utilities/hawqssh.html.md.erb b/reference/cli/admin_utilities/hawqssh.html.md.erb
new file mode 100644
index 0000000..e572bf5
--- /dev/null
+++ b/reference/cli/admin_utilities/hawqssh.html.md.erb
@@ -0,0 +1,96 @@
+---
+title: hawq ssh
+---
+
+Provides SSH access to multiple hosts at once.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+hawq ssh { -f hostfile_hawqssh | - h hostname [-h hostname ...] } [-v] [-e]
+ [bash_command]
+
+hawq ssh -?
+
+hawq ssh --version
+```
+
+## Description<a id="topic1__section3"></a>
+
+The `hawq ssh` utility allows you to run bash shell commands on multiple hosts at once using SSH (secure shell). You can execute a single command by specifying it on the command-line, or omit the command to enter into an interactive command-line session.
+
+To specify the hosts involved in the SSH session, use the `-f` option to specify a file containing a list of host names, or use the `-h` option to name single host names on the command-line. At least one host name (`-h`) or a host file (`-f`) is required. Note that the current host is ***not*** included in the session by default — to include the local host, you must explicitly declare it in the list of hosts involved in the session.
+
+Before using `hawq ssh`, you must have a trusted host setup between the hosts involved in the SSH session. You can use the utility `hawq ssh-exkeys` to update the known host files and exchange public keys between hosts if you have not done so already.
+
+If you do not specify a command on the command-line, `hawq ssh` will go into interactive mode. At the `hawq ssh` command prompt (`=>`), you can enter a command as you would in a regular bash terminal command-line, and the command will be executed on all hosts involved in the session. To end an interactive session, press `CTRL`+`D` on the keyboard or type `exit` or `quit`.
+
+If a user name is not specified in the host file, `hawq ssh` will execute commands as the currently logged in user. To determine the currently logged in user, do a `whoami` command. By default, `hawq ssh` goes to `$HOME` of the session user on the remote hosts after login. To ensure commands are executed correctly on all remote hosts, you should always enter absolute paths.
+
+## Options<a id="topic1__section4"></a>
+
+ *bash\_command*
+A bash shell command to execute on all hosts involved in this session (optionally enclosed in quotes). If not specified, `hawq ssh` will start an interactive session.
+
+-e (echo)
+Optional. Echoes the commands passed to each host and their resulting output while running in non-interactive mode.
+
+-f *hostfile\_hawqssh*
+Specifies the name of a file that contains a list of hosts that will participate in this SSH session. The host name is required, and you can optionally specify an alternate user name and/or SSH port number per host. The syntax of the host file is one host per line as follows:
+
+``` pre
+[username@]hostname[:ssh_port]
+```
+
+-h *hostname*
+Specifies a single host name that will participate in this SSH session. You can use the `-h` option multiple times to specify multiple host names.
+
+-v (verbose mode)
+Optional. Reports additional messages in addition to the command output when running in non-interactive mode.
+
+--version
+Displays the version of this utility.
+
+-? (help)
+Displays the online help.
+
+## Examples<a id="topic1__section5"></a>
+
+Start an interactive group SSH session with all hosts listed in the file `hostfile_hawqssh`:
+
+``` pre
+$ hawq ssh -f hostfile_hawqssh
+```
+
+At the `hawq ssh` interactive command prompt, run a shell command on all the hosts involved in this session.
+
+``` pre
+=> ls -a /data/path-to-masterdd/*
+```
+
+Exit an interactive session:
+
+``` pre
+=> exit
+=> quit
+```
+
+Start a non-interactive group SSH session with the hosts named `sdw1` and `sdw2` and pass a file containing several commands named `command_file` to `hawq ssh`:
+
+``` pre
+$ hawq ssh -h sdw1 -h sdw2 -v -e < command_file
+```
+
+Execute single commands in non-interactive mode on hosts `sdw2` and `localhost`:
+
+``` pre
+$ hawq ssh -h sdw2 -h localhost -v -e 'ls -a /data/primary/*'
+$ hawq ssh -h sdw2 -h localhost -v -e 'echo $GPHOME'
+$ hawq ssh -h sdw2 -h localhost -v -e 'ls -1 | wc -l'
+```
+
+## See Also
+
+[hawq ssh-exkeys](hawqssh-exkeys.html#topic1), [hawq scp](hawqscp.html#topic1)
+
+
diff --git a/reference/cli/admin_utilities/hawqstart.html.md.erb b/reference/cli/admin_utilities/hawqstart.html.md.erb
new file mode 100644
index 0000000..95a7d35
--- /dev/null
+++ b/reference/cli/admin_utilities/hawqstart.html.md.erb
@@ -0,0 +1,118 @@
+---
+title: hawq start
+---
+
+Starts a HAWQ system.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+hawq start object [-l| --logdir] [-q| --quiet]
+ [-v|--verbose] [-m|--masteronly] [-t|--timeout timeout_seconds]
+ [-R | --restrict] [-U | --special-mode maintenance]
+ [--ignore_bad-hosts] cluster | allsegments
+
+```
+
+``` pre
+hawq start -? | -h | --help
+
+hawq start --version
+```
+
+## Description<a id="topic1__section3"></a>
+
+The `hawq start` utility is used to start the HAWQ server processes. When you start a HAWQ system, you are actually starting several `postgres` database server listener processes at once (the master and all of the segment instances). The `hawq start` utility handles the startup of the individual instances. Each instance is started in parallel.
+
+The *object* in the command specifies what entity should be started: e.g. a cluster, a segment, the master node, standby node, or all segments in the cluster.
+
+The first time an administrator runs `hawq start cluster`, the utility creates a static hosts cache file named `$GPHOME/etc/slaves` to store the segment host names. Subsequently, the utility uses this list of hosts to start the system more efficiently. The utility will create a new hosts cache file at each startup.
+
+The `hawq start master` command starts only the HAWQ master, without segment or standby nodes. These can be started later, using `hawq start segment` and/or `hawq start standby`.
+
+**Note:** Typically you should always use `hawq start cluster` or `hawq restart cluster` to start the cluster. If you do end up using `hawq start standby|master|segment` to start nodes individually, make sure you always start the standby before the active master. Otherwise, the standby can become unsynchronized with the active master.
+
+Before you can start a HAWQ system, you must have initialized the system or node by using `hawq init <object>` first.
+
+## Objects
+
+cluster
+Start a HAWQ cluster.
+
+master
+Start HAWQ master.
+
+segment
+Start a local segment node.
+
+standby
+Start a HAWQ standby.
+
+allsegments
+Start all segments.
+
+## Options<a id="topic1__section4"></a>
+
+-l --logdir *logfile\_directory*
+Specifies the log directory for logs of the management tools. The default is `~/hawq/Adminlogs/`.
+
+-q --quiet
+Run in quiet mode. Command output is not displayed on the screen, but is still written to the log file.
+
+-v --verbose
+Displays detailed status, progress and error messages output by the utility.
+
+-m --masteronly
+Optional. Starts the HAWQ master instance only, in utility mode, which may be useful for maintenance tasks. This mode only allows connections to the master in utility mode. For example:
+
+``` pre
+PGOPTIONS='-c gp_role=utility' psql
+```
+
+-R --restrict (restricted mode)
+Starts HAWQ in restricted mode (only database superusers are allowed to connect).
+
+-t --timeout *timeout\_seconds*
+Specifies a timeout in seconds to wait for a segment instance to start up. If a segment instance was shutdown abnormally (due to power failure or killing its `postgres` database listener process, for example), it may take longer to start up due to the database recovery and validation process. If not specified, the default timeout is 60 seconds.
+
+-U --special-mode maintenance
+(Superuser only) Start HAWQ in \[maintenance | upgrade\] mode. In maintenance mode, the `gp_maintenance_conn` parameter is set.
+
+--ignore\_bad\_hosts cluster | allsegments
+Overrides copying configuration files to a host on which SSH validation fails. If ssh to a skipped host is reestablished, make sure the configuration files are re-synched once it is reachable.
+
+-? | -h | --help (help)
+Displays the online help.
+
+--version (show utility version)
+Displays the version of this utility.
+
+## Examples<a id="topic1__section5"></a>
+
+Start a HAWQ system:
+
+``` pre
+hawq start cluster
+```
+
+Start a HAWQ master in maintenance mode:
+
+``` pre
+hawq start master -m
+```
+
+Start a HAWQ system in restricted mode (only allow superuser connections):
+
+``` pre
+hawq start cluster -R
+```
+
+Start the HAWQ master instance only and connect in utility mode:
+
+``` pre
+hawq start master -m PGOPTIONS='-c gp_session_role=utility' psql
+```
+
+## See Also<a id="topic1__section6"></a>
+
+[hawq stop](hawqstop.html#topic1), [hawq init](hawqinit.html#topic1)
diff --git a/reference/cli/admin_utilities/hawqstate.html.md.erb b/reference/cli/admin_utilities/hawqstate.html.md.erb
new file mode 100644
index 0000000..ece0af3
--- /dev/null
+++ b/reference/cli/admin_utilities/hawqstate.html.md.erb
@@ -0,0 +1,58 @@
+---
+title: hawq state
+---
+
+Shows the status of a running HAWQ system.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+hawq state [-b] [-l log_directory] [-v | -q]
+ [-d master_data_directory] [--hawqhome]
+hawq state -? | -h | --help
+```
+
+## Description<a id="topic1__section3"></a>
+
+The `hawq state` utility displays information about a running HAWQ instance. A HAWQ system is comprised of multiple PostgreSQL database instances (segments) spanning multiple machines, and the `hawq state` utility can provide additional status information, such as:
+
+- Total segment count.
+- Which segments are down.
+- Master and segment configuration information (hosts, data directories, etc.).
+- The ports used by the system.
+- Whether a standby master is present, and if it is active.
+
+## Options<a id="topic1__section4"></a>
+
+-b (brief status)
+Optional. Display a brief summary of the state of the HAWQ system. This is the default option.
+
+-d *master\_data\_directory* --datadir=MASTER\_DIR
+Status of the master data directory. If not specified, the value set for `$MASTER_DATA_DIRECTORY` will be used.
+
+--hawqhome=HAWQ\_HOME
+Display details of the designated home data directory.
+
+-l *logfile\_directory*
+Displays calls to the log file. Defaults to `~/hawq/AdminLogs`.
+
+-q (no screen output)
+Optional. Run in quiet mode. Except for warning messages, command output is not displayed on the screen. However, this information is still written to the log file.
+
+-v (verbose output)
+Optional. Displays error messages and outputs detailed status and progress information.
+
+-? | -h | --help (help)
+Displays the online help.
+
+## Examples<a id="topic1__section6"></a>
+
+Show brief status information of a HAWQ system:
+
+``` pre
+hawq state -b
+```
+
+## See Also<a id="topic1__section7"></a>
+
+[hawq start](hawqstart.html#topic1), [gplogfilter](gplogfilter.html#topic1)
diff --git a/reference/cli/admin_utilities/hawqstop.html.md.erb b/reference/cli/admin_utilities/hawqstop.html.md.erb
new file mode 100644
index 0000000..cb403ec
--- /dev/null
+++ b/reference/cli/admin_utilities/hawqstop.html.md.erb
@@ -0,0 +1,110 @@
+---
+title: hawq stop
+---
+
+Stops or restarts a HAWQ system.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+hawq stop object [-a]
+ [-M|--mode smart | fast | immediate] [-t timeout_seconds]
+ [-l logfile_directory] [-v|--verbose | -q|--quiet]
+ [--ignore_bad-hosts] cluster | allsegments
+hawq stop object -u|--reload [-l logfile_directory]
+
+hawq stop --version
+
+hawq stop -? | -h | --help
+```
+
+## Description<a id="topic1__section3"></a>
+
+The `hawq stop` utility is used to stop the database servers that comprise a HAWQ system. When you stop a HAWQ system, you are actually stopping several `postgres` database server processes at once (the master and all of the segment instances). The `hawq stop` utility handles the shutdown of the individual instances. Each instance is shutdown in parallel.
+
+By default, you are not allowed to shut down HAWQ if there are any client connections to the database. Use the `-M fast` option to roll back all in progress transactions and terminate any connections before shutting down. If there are any transactions in progress, the default behavior is to wait for them to commit before shutting down.
+
+With the `-u` option, the utility uploads changes made to the master `pg_hba.conf` file or to *runtime* configuration parameters in the master `hawq-site.xml` file without interruption of service. Note that any active sessions will not pick up the changes until they reconnect to the database.
+
+## Objects
+
+cluster
+Stop a HAWQ cluster.
+
+master
+Shuts down a HAWQ master instance that was started in maintenance mode.
+
+segment
+Stop a local segment node.
+
+standby
+Stop the HAWQ standby master process.
+
+allsegments
+Stop all segments.
+
+## Options<a id="topic1__section4"></a>
+
+-a --prompt
+Do not prompt the user for confirmation before executing.
+
+-l --logdir*logfile\_directory*
+The directory to write the log file. The default is `~/hawq/Adminlogs/`.
+
+-M smart | fast | immediate
+Smart shutdown is the default. Shutdown fails with a warning message, if active connections are found.
+
+Fast shut down interrupts and rolls back any transactions currently in progress .
+
+Immediate shutdown aborts transactions in progress and kills all `postgres` processes without allowing the database server to complete transaction processing or clean up any temporary or in-process work files. Because of this, immediate shutdown is not recommended. In some instances, it can cause database corruption that requires manual recovery.
+
+-q --quiet
+Run in quiet mode. Command output is not displayed on the screen, but is still written to the log file.
+
+-t --timeout*timeout\_seconds*
+Specifies a timeout threshold (in seconds) to wait for a segment instance to shutdown. If a segment instance does not shut down in the specified number of seconds, `hawq stop` displays a message indicating that one or more segments are still in the process of shutting down and that you cannot restart HAWQ until the segment instance(s) are stopped. This option is useful in situations where `hawq stop` is executed and there are very large transactions that need to rollback. These large transactions can take over a minute to rollback and surpass the default timeout period of 600 seconds.
+
+--ignore\_bad\_hosts cluster | allsegments
+Overrides copying configuration files to a host on which SSH validation fails. If ssh to a skipped host is reestablished, make sure the configuration files are re-synched once it is reachable.
+
+-u --reload
+This option reloads configuration parameter values without restarting the HAWQ cluster.
+
+-v --verbose
+Displays detailed status, progress and error messages output by the utility.
+
+-? | -h | --help (help)
+Displays the online help.
+
+--version (show utility version)
+Displays the version of this utility.
+
+## Examples<a id="topic1__section5"></a>
+
+Stop a HAWQ system in smart mode:
+
+``` pre
+hawq stop cluster -M smart
+```
+
+Stop a HAWQ system in fast mode:
+
+``` pre
+hawq stop cluster -M fast
+```
+
+Stop a master instance that was started in maintenance mode:
+
+``` pre
+hawq stop master -m
+```
+
+Reload the `hawq-site.xml` and `pg_hba.conf` files after making configuration changes but do not shutdown the HAWQ array:
+
+``` pre
+hawq stop cluster -u
+```
+
+## See Also<a id="topic1__section6"></a>
+
+[hawq start](hawqstart.html#topic1)
diff --git a/reference/cli/client_utilities/createdb.html.md.erb b/reference/cli/client_utilities/createdb.html.md.erb
new file mode 100644
index 0000000..3e9fad5
--- /dev/null
+++ b/reference/cli/client_utilities/createdb.html.md.erb
@@ -0,0 +1,78 @@
+---
+title: createdb
+---
+
+Creates a new database.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+createdb [connection_option ...] [-D tablespace] [-E encoding]
+ [-O owner] [-T template] [-e] [dbname ['description']]
+
+createdb --help
+
+createdb --version
+```
+
+## Description<a id="topic1__section3"></a>
+
+`createdb` creates a new database in a HAWQ system.
+
+Normally, the database user who executes this command becomes the owner of the new database. However a different owner can be specified via the `-O` option, if the executing user has appropriate privileges.
+
+`createdb` is a wrapper around the SQL command `CREATE DATABASE`.
+
+## Options<a id="topic1__section4"></a>
+
+dbname
+The name of the database to be created. The name must be unique among all other databases in the HAWQ system. If not specified, reads from the environment variable `PGDATABASE`, then `PGUSER` or defaults to the current system user.
+
+description
+A comment to be associated with the newly created database. Descriptions containing white space must be enclosed in quotes.
+
+-D *tablespace* | --tablespace *tablespace*
+The default tablespace for the database.
+
+-e echo
+Echo the commands that createdb generates and sends to the server.
+
+-E *encoding* | --encoding *encoding*
+Character set encoding to use in the new database. Specify a string constant (such as `'UTF8'`), an integer encoding number, or `DEFAULT` to use the default encoding.
+
+-O *owner* | --owner *owner*
+The name of the database user who will own the new database. Defaults to the user executing this command.
+
+-T *template* | --template *template*
+The name of the template from which to create the new database. Defaults to `template1`.
+
+**Connection Options**
+
+-h *host* | --host *host*
+The host name of the machine on which the HAWQ master database server is running. If not specified, reads from the environment variable `PGHOST` or defaults to localhost.
+
+-p *port* | --port *port*
+The TCP port on which the HAWQ master database server is listening for connections. If not specified, reads from the environment variable `PGPORT` or defaults to 5432.
+
+-U *username* | --username *username*
+The database role name to connect as. If not specified, reads from the environment variable `PGUSER` or defaults to the current system role name.
+
+-w | --no-password
+Never issue a password prompt. If the server requires password authentication and a password is not available by other means such as a `.pgpass` file, the connection attempt will fail. This option can be useful in batch jobs and scripts where no user is present to enter a password.
+
+-W | --password
+Force a password prompt.
+
+## Examples<a id="topic1__section6"></a>
+
+To create the database `test` using the default options:
+
+``` pre
+createdb test
+```
+
+To create the database `demo` using the HAWQ master on host `gpmaster`, port `54321`, using the `LATIN1` encoding scheme:
+
+``` pre
+createdb -p 54321 -h gpmaster -E LATIN1 demo
+```
diff --git a/reference/cli/client_utilities/createuser.html.md.erb b/reference/cli/client_utilities/createuser.html.md.erb
new file mode 100644
index 0000000..3eb24e1
--- /dev/null
+++ b/reference/cli/client_utilities/createuser.html.md.erb
@@ -0,0 +1,124 @@
+---
+title: createuser
+---
+
+Creates a new database role.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+createuser [connection_option ...] [role_attribute ...] [-e] role_name
+
+createuser --help
+
+createuser --version
+```
+
+## Description<a id="topic1__section3"></a>
+
+`createuser` creates a new HAWQ /> role. You must be a superuser or have the `CREATEROLE` privilege to create new roles. You must connect to the database as a superuser to create new superusers.
+
+Superusers can bypass all access permission checks within the database, so superuser privileges should not be granted lightly.
+
+`createuser` is a wrapper around the SQL command `CREATE ROLE`.
+
+## Options<a id="topic1__section4"></a>
+
+role\_name
+The name of the role to be created. This name must be different from all existing roles in this HAWQ installation.
+
+-c *number* | --connection-limit *number*
+Set a maximum number of connections for the new role. The default is to set no limit.
+
+-D | --no-createdb
+The new role will not be allowed to create databases. This is the default.
+
+-d | --createdb
+The new role will be allowed to create databases.
+
+-e | --echo
+Echo the commands that `createuser` generates and sends to the server.
+
+-E | --encrypted
+Encrypts the role's password stored in the database. If not specified, the default password behavior is used.
+
+-i | --inherit
+The new role will automatically inherit privileges of roles it is a member of. This is the default.
+
+-I | --no-inherit
+The new role will not automatically inherit privileges of roles it is a member of.
+
+-l | --login
+The new role will be allowed to log in to HAWQ. This is the default.
+
+-L | --no-login
+The new role will not be allowed to log in (a group-level role).
+
+-N | --unencrypted
+Does not encrypt the role's password stored in the database. If not specified, the default password behavior is used.
+
+-P | --pwprompt
+If given, `createuser` will issue a prompt for the password of the new role. This is not necessary if you do not plan on using password authentication.
+
+-r | --createrole
+The new role will be allowed to create new roles (`CREATEROLE` privilege).
+
+-R | --no-createrole
+The new role will not be allowed to create new roles. This is the default.
+
+-s | --superuser
+The new role will be a superuser.
+
+-S | --no-superuser
+The new role will not be a superuser. This is the default.
+
+**Connection Options**
+
+-h *host* | --host *host*
+The host name of the machine on which the HAWQ master database server is running. If not specified, reads from the environment variable `PGHOST` or defaults to localhost.
+
+-p *port* | --port *port*
+The TCP port on which the HAWQ master database server is listening for connections. If not specified, reads from the environment variable `PGPORT` or defaults to 5432.
+
+-U *username* | --username *username*
+The database role name to connect as. If not specified, reads from the environment variable `PGUSER` or defaults to the current system role name.
+
+-w | --no-password
+Never issue a password prompt. If the server requires password authentication and a password is not available by other means such as a `.pgpass` file, the connection attempt will fail. This option can be useful in batch jobs and scripts where no user is present to enter a password.
+
+-W | --password
+Force a password prompt.
+
+## Examples<a id="topic1__section6"></a>
+
+Create a role named `joe` using the default options:
+
+``` pre
+createuser joe
+Shall the new role be a superuser? (y/n) n
+Shall the new role be allowed to create databases? (y/n) n
+Shall the new role be allowed to create more new roles? (y/n) n
+CREATE ROLE
+```
+
+To create the same role `joe` using connection options and avoiding the prompts and taking a look at the underlying command:
+
+``` pre
+createuser -h masterhost -p 54321 -S -D -R -e joe
+CREATE ROLE joe NOSUPERUSER NOCREATEDB NOCREATEROLE INHERIT
+LOGIN;
+CREATE ROLE
+```
+
+To create the role `joe` as a superuser, and assign password `admin123` immediately:
+
+``` pre
+createuser -P -s -e joe
+Enter password for new role: admin123
+Enter it again: admin123
+CREATE ROLE joe PASSWORD 'admin123' SUPERUSER CREATEDB
+CREATEROLE INHERIT LOGIN;
+CREATE ROLE
+```
+
+In the above example, the new password is not actually echoed when typed, but we show what was typed for clarity. However the password will appear in the echoed command, as illustrated if the `-e` option is used.
diff --git a/reference/cli/client_utilities/dropdb.html.md.erb b/reference/cli/client_utilities/dropdb.html.md.erb
new file mode 100644
index 0000000..2044a45
--- /dev/null
+++ b/reference/cli/client_utilities/dropdb.html.md.erb
@@ -0,0 +1,67 @@
+---
+title: dropdb
+---
+
+Removes a database.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+dropdb [connection_option ...] [-e] [-i] dbname
+
+dropdb --help
+
+dropdb --version
+```
+
+## Description<a id="topic1__section3"></a>
+
+`dropdb` destroys an existing database. The user who executes this command must be a superuser or the owner of the database being dropped.
+
+`dropdb` is a wrapper around the SQL command `DROP DATABASE`.
+
+## Options<a id="topic1__section4"></a>
+
+*dbname*
+The name of the database to be removed.
+
+-e | --echo
+Echo the commands that `dropdb` generates and sends to the server.
+
+-i | --interactive
+Issues a verification prompt before doing anything destructive.
+
+**Connection Options**
+
+-h *host* | --host *host*
+The host name of the machine on which the HAWQ master database server is running. If not specified, reads from the environment variable `PGHOST` or defaults to localhost.
+
+-p *port* | --port *port*
+The TCP port on which the HAWQ master database server is listening for connections. If not specified, reads from the environment variable `PGPORT` or defaults to 5432.
+
+-U *username* | --username *username*
+The database role name to connect as. If not specified, reads from the environment variable `PGUSER` or defaults to the current system role name.
+
+-w | --no-password
+Never issue a password prompt. If the server requires password authentication and a password is not available by other means such as a `.pgpass` file, the connection attempt will fail. This option can be useful in batch jobs and scripts where no user is present to enter a password.
+
+-W | --password
+Force a password prompt.
+
+## Examples<a id="topic1__section6"></a>
+
+To destroy the database named `demo` using default connection parameters:
+
+``` pre
+dropdb demo
+```
+
+To destroy the database named `demo` using connection options, with verification, and a peek at the underlying command:
+
+``` pre
+dropdb -p 54321 -h masterhost -i -e demo
+Database "demo" will be permanently deleted.
+Are you sure? (y/n) y
+DROP DATABASE "demo"
+DROP DATABASE
+```
diff --git a/reference/cli/client_utilities/dropuser.html.md.erb b/reference/cli/client_utilities/dropuser.html.md.erb
new file mode 100644
index 0000000..2ae81a2
--- /dev/null
+++ b/reference/cli/client_utilities/dropuser.html.md.erb
@@ -0,0 +1,66 @@
+---
+title: dropuser
+---
+
+Removes a database role.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+dropuser [connection_option ...] [-e] [-i] role_name
+
+dropuser --help
+
+dropuser --version
+```
+
+## Description<a id="topic1__section3"></a>
+
+`dropuser` removes an existing role from HAWQ. Only superusers and users with the `CREATEROLE` privilege can remove roles. To remove a superuser role, you must yourself be a superuser.
+
+`dropuser` is a wrapper around the SQL command `DROP ROLE`.
+
+## Options<a id="topic1__section4"></a>
+
+*role\_name*
+The name of the role to be removed. You will be prompted for a name if not specified on the command line.
+
+-e | --echo
+Echo the commands that `dropuser` generates and sends to the server.
+
+-i | --interactive
+Prompt for confirmation before actually removing the role.
+
+**Connection Options**
+
+-h *host* | --host *host*
+The host name of the machine on which the HAWQ master database server is running. If not specified, reads from the environment variable `PGHOST` or defaults to localhost.
+
+-p *port* | --port *port*
+The TCP port on which the HAWQ master database server is listening for connections. If not specified, reads from the environment variable `PGPORT` or defaults to 5432.
+
+-U *username* | --username *username*
+The database role name to connect as. If not specified, reads from the environment variable `PGUSER` or defaults to the current system role name.
+
+-w | --no-password
+Never issue a password prompt. If the server requires password authentication and a password is not available by other means such as a `.pgpass` file, the connection attempt will fail. This option can be useful in batch jobs and scripts where no user is present to enter a password.
+
+-W | --password
+Force a password prompt.
+
+## Examples<a id="topic1__section6"></a>
+
+To remove the role `joe` using default connection options:
+
+``` pre
+dropuser joe
+```
+
+To remove the role `joe` using connection options, with verification, and a peek at the underlying command:
+
+``` pre
+dropuser -p 54321 -h masterhost -i -e joe
+Role "joe" will be permanently removed.
+Are you sure? (y/n) y
+DROP ROLE "joe"
+```
diff --git a/reference/cli/client_utilities/pg_dump.html.md.erb b/reference/cli/client_utilities/pg_dump.html.md.erb
new file mode 100644
index 0000000..b3b8644
--- /dev/null
+++ b/reference/cli/client_utilities/pg_dump.html.md.erb
@@ -0,0 +1,199 @@
+---
+title: pg_dump
+---
+
+Extracts a database into a single script file or other archive file.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+pg_dump [connection_option ...] [dump_option ...] dbname
+```
+
+## Description<a id="topic1__section3"></a>
+
+`pg_dump` is a standard PostgreSQL utility for backing up a database, and is also supported in HAWQ. It creates a single (non-parallel) dump file.
+
+Use `pg_dump` if you are migrating your data to another database vendor's system, or to another HAWQ system with a different segment configuration (for example, if the system you are migrating to has greater or fewer segment instances). To restore, you must use the corresponding [pg\_restore](pg_restore.html#topic1) utility (if the dump file is in archive format), or you can use a client program such as [psql](psql.html#topic1) (if the dump file is in plain text format).
+
+Since `pg_dump` is compatible with regular PostgreSQL, it can be used to migrate data into HAWQ. The `pg_dump` utility in HAWQ is very similar to the PostgreSQL `pg_dump` utility, with the following exceptions and limitations:
+
+- If using `pg_dump` to backup a HAWQ database, keep in mind that the dump operation can take a long time (several hours) for very large databases. Also, you must make sure you have sufficient disk space to create the dump file.
+- If you are migrating data from one HAWQ system to another, use the `--gp-syntax` command-line option to include the `DISTRIBUTED BY` clause in `CREATE TABLE` statements. This ensures that HAWQ table data is distributed with the correct distribution key columns upon restore.
+
+`pg_dump` makes consistent backups even if the database is being used concurrently. `pg_dump` does not block other users accessing the database (readers or writers).
+
+When used with one of the archive file formats and combined with `pg_restore`, `pg_dump` provides a flexible archival and transfer mechanism. `pg_dump` can be used to backup an entire database, then `pg_restore `can be used to examine the archive and/or select which parts of the database are to be restored. The most flexible output file format is the *custom* format (`-Fc`). It allows for selection and reordering of all archived items, and is compressed by default. The tar format (`-Ft`) is not compressed and it is not possible to reorder data when loading, but it is otherwise quite flexible. It can be manipulated with standard UNIX tools such as `tar`.
+
+## Options<a id="topic1__section4"></a>
+
+dbname
+Specifies the name of the database to be dumped. If this is not specified, the environment variable `PGDATABASE` is used. If that is not set, the user name specified for the connection is used.
+
+**Dump Options**
+
+-a | --data-only
+Dump only the data, not the schema (data definitions). This option is only meaningful for the plain-text format. For the archive formats, you may specify the option when you call [pg\_restore](pg_restore.html#topic1).
+
+-b | --blobs
+Include large objects in the dump. This is the default behavior except when `--schema`, `--table`, or `--schema-only` is specified, so the `-b` switch is only useful to add large objects to selective dumps.
+
+-c | --clean
+Adds commands to the text output file to clean (drop) database objects prior to (the commands for) creating them. Note that objects are not dropped before the dump operation begins, but `DROP` commands are added to the DDL dump output files so that when you use those files to do a restore, the `DROP` commands are run prior to the `CREATE` commands. This option is only meaningful for the plain-text format. For the archive formats, you may specify the option when you call [pg\_restore](pg_restore.html#topic1).
+
+-C | --create
+Begin the output with a command to create the database itself and reconnect to the created database. (With a script of this form, it doesn't matter which database you connect to before running the script.) This option is only meaningful for the plain-text format. For the archive formats, you may specify the option when you call [pg\_restore](pg_restore.html#topic1).
+
+-d | --inserts
+Dump data as `INSERT` commands (rather than `COPY`). This will make restoration very slow; it is mainly useful for making dumps that can be loaded into non-PostgreSQL-based databases. Also, since this option generates a separate command for each row, an error in reloading a row causes only that row to be lost rather than the entire table contents. Note that the restore may fail altogether if you have rearranged column order. The `-D` option is safe against column order changes, though even slower.
+
+-D | --column-inserts | --attribute-inserts
+Dump data as `INSERT` commands with explicit column names `(INSERT INTO` *table*`(`*column*`, ...) VALUES ...)`. This will make restoration very slow; it is mainly useful for making dumps that can be loaded into non-PostgreSQL-based databases. Also, since this option generates a separate command for each row, an error in reloading a row causes only that row to be lost rather than the entire table contents.
+
+-E *encoding* | --encoding=*encoding*
+Create the dump in the specified character set encoding. By default, the dump is created in the database encoding. (Another way to get the same result is to set the `PGCLIENTENCODING` environment variable to the desired dump encoding.)
+
+-f *file* | --file=*file*
+Send output to the specified file. If this is omitted, the standard output is used.
+
+-F p|c|t | --format=plain|custom|tar
+Selects the format of the output. format can be one of the following:
+
+p | plain — Output a plain-text SQL script file (the default).
+
+c | custom — Output a custom archive suitable for input into [pg\_restore](pg_restore.html#topic1). This is the most flexible format in that it allows reordering of loading data as well as object definitions. This format is also compressed by default.
+
+t | tar — Output a tar archive suitable for input into [pg\_restore](pg_restore.html#topic1). Using this archive format allows reordering and/or exclusion of database objects at the time the database is restored. It is also possible to limit which data is reloaded at restore time.
+
+-i | --ignore-version
+Ignore version mismatch between `pg_dump` and the database server. `pg_dump` can dump from servers running previous releases of HAWQ (or PostgreSQL), but very old versions may not be supported anymore. Use this option if you need to override the version check.
+
+-n *schema* | --schema=*schema*
+Dump only schemas matching the schema pattern; this selects both the schema itself, and all its contained objects. When this option is not specified, all non-system schemas in the target database will be dumped. Multiple schemas can be selected by writing multiple `-n` switches. Also, the schema parameter is interpreted as a pattern according to the same rules used by `psql`'s` \d` commands, so multiple schemas can also be selected by writing wildcard characters in the pattern. When using wildcards, be careful to quote the pattern if needed to prevent the shell from expanding the wildcards.
+
+Note: When -n is specified, `pg_dump` makes no attempt to dump any other database objects that the selected schema(s) may depend upon. Therefore, there is no guarantee that the results of a specific-schema dump can be successfully restored by themselves into a clean database.
+
+**Note:** Non-schema objects such as blobs are not dumped when `-n` is specified. You can add blobs back to the dump with the `--blobs` switch.
+
+-N *schema* | --exclude-schema=*schema*
+Do not dump any schemas matching the schema pattern. The pattern is interpreted according to the same rules as for `-n`. `-N` can be given more than once to exclude schemas matching any of several patterns. When both `-n` and `-N` are given, the behavior is to dump just the schemas that match at least one `-n` switch but no `-N` switches. If `-N` appears without `-n`, then schemas matching `-N` are excluded from what is otherwise a normal dump.
+
+-o | --oids
+Dump object identifiers (OIDs) as part of the data for every table. Use of this option is not recommended for files that are intended to be restored into HAWQ.
+
+-O | --no-owner
+Do not output commands to set ownership of objects to match the original database. By default, `pg_dump` issues `ALTER OWNER` or `SET SESSION AUTHORIZATION` statements to set ownership of created database objects. These statements will fail when the script is run unless it is started by a superuser (or the same user that owns all of the objects in the script). To make a script that can be restored by any user, but will give that user ownership of all the objects, specify `-O`. This option is only meaningful for the plain-text format. For the archive formats, you may specify the option when you call [pg\_restore](pg_restore.html#topic1).
+
+-s | --schema-only
+Dump only the object definitions (schema), not data.
+
+-S *username* | --superuser=*username*
+Specify the superuser user name to use when disabling triggers. This is only relevant if `--disable-triggers` is used. It is better to leave this out, and instead start the resulting script as a superuser.
+
+**Note:** HAWQ does not support user-defined triggers.
+
+-t *table* | --table=*table*
+Dump only tables (or views or sequences) matching the table pattern. Specify the table in the format `schema.table`.
+
+Multiple tables can be selected by writing multiple `-t` switches. Also, the table parameter is interpreted as a pattern according to the same rules used by `psql`'s `\d` commands, so multiple tables can also be selected by writing wildcard characters in the pattern. When using wildcards, be careful to quote the pattern if needed to prevent the shell from expanding the wildcards. The `-n` and `-N` switches have no effect when `-t` is used, because tables selected by `-t` will be dumped regardless of those switches, and non-table objects will not be dumped.
+
+**Note:** When `-t` is specified, `pg_dump` makes no attempt to dump any other database objects that the selected table(s) may depend upon. Therefore, there is no guarantee that the results of a specific-table dump can be successfully restored by themselves into a clean database.
+Also, `-t` cannot be used to specify a child table partition. To dump a partitioned table, you must specify the parent table name.
+
+-T *table* | --exclude-table=*table*
+Do not dump any tables matching the table pattern. The pattern is interpreted according to the same rules as for `-t`. `-T` can be given more than once to exclude tables matching any of several patterns. When both `-t` and `-T` are given, the behavior is to dump just the tables that match at least one `-t` switch but no `-T` switches. If `-T` appears without `-t`, then tables matching `-T` are excluded from what is otherwise a normal dump.
+
+-v | --verbose
+Specifies verbose mode. This will cause `pg_dump` to output detailed object comments and start/stop times to the dump file, and progress messages to standard error.
+
+-x | --no-privileges | --no-acl
+Prevent dumping of access privileges (`GRANT/REVOKE` commands).
+
+--disable-dollar-quoting
+This option disables the use of dollar quoting for function bodies, and forces them to be quoted using SQL standard string syntax.
+
+--disable-triggers
+This option is only relevant when creating a data-only dump. It instructs `pg_dump` to include commands to temporarily disable triggers on the target tables while the data is reloaded. Use this if you have triggers on the tables that you do not want to invoke during data reload. The commands emitted for `--disable-triggers` must be done as superuser. So, you should also specify a superuser name with `-S`, or preferably be careful to start the resulting script as a superuser. This option is only meaningful for the plain-text format. For the archive formats, you may specify the option when you call [pg\_restore](pg_restore.html#topic1).
+
+**Note:** HAWQ does not support user-defined triggers.
+
+--use-set-session-authorization
+Output SQL-standard `SET SESSION AUTHORIZATION` commands instead of `ALTER OWNER` commands to determine object ownership. This makes the dump more standards compatible, but depending on the history of the objects in the dump, may not restore properly. A dump using `SET SESSION AUTHORIZATION` will require superuser privileges to restore correctly, whereas `ALTER OWNER` requires lesser privileges.
+
+--gp-syntax | --no-gp-syntax
+Use `--gp-syntax` to dump HAWQ syntax in the `CREATE TABLE` statements. This allows the distribution policy (`DISTRIBUTED BY` or `DISTRIBUTED RANDOMLY` clauses) of a HAWQ table to be dumped, which is useful for restoring into other HAWQ systems. The default is to include HAWQ syntax when connected to a HAWQ system, and to exclude it when connected to a regular PostgreSQL system.
+
+-Z 0..9 | --compress=0..9
+Specify the compression level to use in archive formats that support compression. Currently only the *custom* archive format supports compression.
+
+**Connection Options**
+
+-h *host*| --host *host*
+The host name of the machine on which the HAWQ master database server is running. If not specified, reads from the environment variable `PGHOST` or defaults to localhost.
+
+-p *port*| --port *port*
+The TCP port on which the HAWQ master database server is listening for connections. If not specified, reads from the environment variable `PGPORT` or defaults to 5432.
+
+-U *username*| --username *username*
+The database role name to connect as. If not specified, reads from the environment variable `PGUSER` or defaults to the current system role name.
+
+-W | --password
+Force a password prompt.
+
+## Notes<a id="topic1__section7"></a>
+
+When a data-only dump is chosen and the option `--disable-triggers` is used, `pg_dump` emits commands to disable triggers on user tables before inserting the data and commands to re-enable them after the data has been inserted. If the restore is stopped in the middle, the system catalogs may be left in the wrong state.
+
+Members of `tar` archives are limited to a size less than 8 GB. (This is an inherent limitation of the `tar` file format.) Therefore this format cannot be used if the textual representation of any one table exceeds that size. The total size of a tar archive and any of the other output formats is not limited, except possibly by the operating system.
+
+The dump file produced by `pg_dump` does not contain the statistics used by the optimizer to make query planning decisions. Therefore, it is wise to run `ANALYZE` after restoring from a dump file to ensure good performance.
+
+## Examples<a id="topic1__section8"></a>
+
+Dump a database called `mydb` into a SQL-script file:
+
+``` pre
+pg_dump mydb > db.sql
+```
+
+To reload such a script into a (freshly created) database named `newdb`:
+
+``` pre
+psql -d newdb -f db.sql
+```
+
+Dump a HAWQ in tar file format and include distribution policy information:
+
+``` pre
+pg_dump -Ft --gp-syntax mydb > db.tar
+```
+
+To dump a database into a custom-format archive file:
+
+``` pre
+pg_dump -Fc mydb > db.dump
+```
+
+To reload an archive file into a (freshly created) database named `newdb`:
+
+``` pre
+pg_restore -d newdb db.dump
+```
+
+**Note:** A warning related to the `gp_enable_column_oriented_table` parameter may appear. If it does, disregard it.
+
+To dump a single table named `mytab`:
+
+``` pre
+pg_dump -t mytab mydb > db.sql
+```
+
+To specify an upper-case or mixed-case name in `-t` and related switches, you need to double-quote the name; else it will be folded to lower case. But double quotes are special to the shell, so in turn they must be quoted. Thus, to dump a single table with a mixed-case name, you need something like:
+
+``` pre
+pg_dump -t '"MixedCaseName"' mydb > mytab.sql
+```
+
+## See Also<a id="topic1__section9"></a>
+
+[pg\_dumpall](pg_dumpall.html#topic1), [pg\_restore](pg_restore.html#topic1), [psql](psql.html#topic1)
diff --git a/reference/cli/client_utilities/pg_dumpall.html.md.erb b/reference/cli/client_utilities/pg_dumpall.html.md.erb
new file mode 100644
index 0000000..fc1b524
--- /dev/null
+++ b/reference/cli/client_utilities/pg_dumpall.html.md.erb
@@ -0,0 +1,131 @@
+---
+title: pg_dumpall
+---
+
+Extracts all databases in a HAWQ system to a single script file or other archive file.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+pg_dumpall [connection_option ...] [dump_option ...]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`pg_dumpall` is a standard PostgreSQL utility for backing up all databases in a HAWQ (or PostgreSQL) instance, and is also supported in HAWQ. It creates a single (non-parallel) dump file.
+
+`pg_dumpall` creates a single script file that contains SQL commands that can be used as input to [psql](psql.html#topic1) to restore the databases. It does this by calling [pg\_dump](pg_dump.html#topic1) for each database. `pg_dumpall` also dumps global objects that are common to all databases. (`pg_dump` does not save these objects.) This currently includes information about database users and groups, and access permissions that apply to databases as a whole.
+
+Since `pg_dumpall` reads tables from all databases you will most likely have to connect as a database superuser in order to produce a complete dump. Also you will need superuser privileges to execute the saved script in order to be allowed to add users and groups, and to create databases.
+
+The SQL script will be written to the standard output. Shell operators should be used to redirect it into a file.
+
+`pg_dumpall` needs to connect several times to the HAWQ master server (once per database). If you use password authentication it is likely to ask for a password each time. It is convenient to have a `~/.pgpass` file in such cases.
+
+## Options<a id="topic1__section4"></a>
+
+**Dump Options**
+
+-a | --data-only
+Dump only the data, not the schema (data definitions). This option is only meaningful for the plain-text format. For the archive formats, you may specify the option when you call [pg\_restore](pg_restore.html#topic1).
+
+-c | --clean
+Output commands to clean (drop) database objects prior to (the commands for) creating them. This option is only meaningful for the plain-text format. For the archive formats, you may specify the option when you call [pg\_restore](pg_restore.html#topic1).
+
+-d | --inserts
+Dump data as `INSERT` commands (rather than `COPY`). This will make restoration very slow; it is mainly useful for making dumps that can be loaded into non-PostgreSQL-based databases. Also, since this option generates a separate command for each row, an error in reloading a row causes only that row to be lost rather than the entire table contents. Note that the restore may fail altogether if you have rearranged column order. The `-D` option is safe against column order changes, though even slower.
+
+-D | --column-inserts | --attribute-inserts
+Dump data as `INSERT` commands with explicit column names `(INSERT INTO table (column, ...) VALUES ...)`. This will make restoration very slow; it is mainly useful for making dumps that can be loaded into non-PostgreSQL-based databases. Also, since this option generates a separate command for each row, an error in reloading a row causes only that row to be lost rather than the entire table contents.
+
+-f | --filespaces
+Dump filespace definitions.
+
+-g | --globals-only
+Dump only global objects (roles and tablespaces), no databases.
+
+-i | --ignore-version
+Ignore version mismatch between [pg\_dump](pg_dump.html#topic1) and the database server. `pg_dump` can dump from servers running previous releases of HAWQ (or PostgreSQL), but very old versions may not be supported anymore. Use this option if you need to override the version check.
+
+-o | --oids
+Dump object identifiers (OIDs) as part of the data for every table. Use of this option is not recommended for files that are intended to be restored into HAWQ.
+
+-O | --no-owner
+Do not output commands to set ownership of objects to match the original database. By default, [pg\_dump](pg_dump.html#topic1) issues `ALTER OWNER` or `SET SESSION AUTHORIZATION` statements to set ownership of created database objects. These statements will fail when the script is run unless it is started by a superuser (or the same user that owns all of the objects in the script). To make a script that can be restored by any user, but will give that user ownership of all the objects, specify `-O`. This option is only meaningful for the plain-text format. For the archive formats, you may specify the option when you call [pg\_restore](pg_restore.html#topic1).
+
+-r | --resource-queues
+Dump resource queue definitions.
+
+-s | --schema-only
+Dump only the object definitions (schema), not data.
+
+-S *username* | --superuser=*username*
+Specify the superuser user name to use when disabling triggers. This is only relevant if `--disable-triggers` is used. It is better to leave this out, and instead start the resulting script as a superuser.
+
+**Note:** HAWQ does not support user-defined triggers.
+
+-v | --verbose
+Specifies verbose mode. This will cause [pg\_dump](pg_dump.html#topic1) to output detailed object comments and start/stop times to the dump file, and progress messages to standard error.
+
+-x | --no-privileges | --no-acl
+Prevent dumping of access privileges (`GRANT/REVOKE` commands).
+
+--disable-dollar-quoting
+This option disables the use of dollar quoting for function bodies, and forces them to be quoted using SQL standard string syntax.
+
+--disable-triggers
+This option is only relevant when creating a data-only dump. It instructs `pg_dumpall` to include commands to temporarily disable triggers on the target tables while the data is reloaded. Use this if you have triggers on the tables that you do not want to invoke during data reload. The commands emitted for `--disable-triggers` must be done as superuser. So, you should also specify a superuser name with `-S`, or preferably be careful to start the resulting script as a superuser.
+
+**Note:** HAWQ does not support user-defined triggers.
+
+--use-set-session-authorization
+Output SQL-standard `SET SESSION AUTHORIZATION` commands instead of `ALTER OWNER` commands to determine object ownership. This makes the dump more standards compatible, but depending on the history of the objects in the dump, may not restore properly. A dump using `SET SESSION AUTHORIZATION` will require superuser privileges to restore correctly, whereas `ALTER OWNER` requires lesser privileges.
+
+--gp-syntax
+Output HAWQ syntax in the `CREATE TABLE` statements. This allows the distribution policy (`DISTRIBUTED BY` or `DISTRIBUTED RANDOMLY` clauses) of a HAWQ table to be dumped, which is useful for restoring into other HAWQ systems.
+
+**Connection Options**
+
+-h *host* | --host *host*
+The host name of the machine on which the HAWQ master database server is running. If not specified, reads from the environment variable `PGHOST` or defaults to `localhost`.
+
+-p *port* | --port *port*
+The TCP port on which the HAWQ master database server is listening for connections. If not specified, reads from the environment variable `PGPORT` or defaults to 5432.
+
+-U *username* | --username *username*
+The database role name to connect as. If not specified, reads from the environment variable `PGUSER` or defaults to the current system role name.
+
+-W | --password
+Force a password prompt.
+
+## Notes<a id="topic1__section7"></a>
+
+Since `pg_dumpall` calls [pg\_dump](pg_dump.html#topic1) internally, some diagnostic messages will refer to `pg_dump`.
+
+Once restored, it is wise to run `ANALYZE` on each database so the query planner has useful statistics. You can also run `vacuumdb -a -z` to analyze all databases.
+
+`pg_dumpall` requires all needed tablespace (filespace) directories to exist before the restore or database creation will fail for databases in non-default locations.
+
+## Examples<a id="topic1__section8"></a>
+
+To dump all databases:
+
+``` pre
+pg_dumpall > db.out
+```
+
+To reload this file:
+
+``` pre
+psql template1 -f db.out
+```
+
+To dump only global objects (including filespaces and resource queues):
+
+``` pre
+pg_dumpall -g -f -r
+```
+
+## See Also<a id="topic1__section9"></a>
+
+[pg\_dump](pg_dump.html#topic1)
diff --git a/reference/cli/client_utilities/pg_restore.html.md.erb b/reference/cli/client_utilities/pg_restore.html.md.erb
new file mode 100644
index 0000000..a4b9de1
--- /dev/null
+++ b/reference/cli/client_utilities/pg_restore.html.md.erb
@@ -0,0 +1,205 @@
+---
+title: pg_restore
+---
+
+Restores a database from an archive file created by `pg_dump`.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+pg_restore [connection_option ...] [restore_option ...] filename
+```
+
+## Description<a id="topic1__section3"></a>
+
+`pg_restore` is a utility for restoring a database from an archive created by [pg\_dump](pg_dump.html#topic1) in one of the non-plain-text formats. It will issue the commands necessary to reconstruct the database to the state it was in at the time it was saved. The archive files also allow `pg_restore` to be selective about what is restored, or even to reorder the items prior to being restored.
+
+`pg_restore` can operate in two modes. If a database name is specified, the archive is restored directly into the database. Otherwise, a script containing the SQL commands necessary to rebuild the database is created and written to a file or standard output. The script output is equivalent to the plain text output format of `pg_dump`. Some of the options controlling the output are therefore analogous to `pg_dump` options.
+
+`pg_restore` cannot restore information that is not present in the archive file. For instance, if the archive was made using the "dump data as `INSERT` commands" option, `pg_restore` will not be able to load the data using `COPY` statements.
+
+## Options<a id="topic1__section4"></a>
+
+ *filename*
+Specifies the location of the archive file to be restored. If not specified, the standard input is used.
+
+**Restore Options**
+
+-a | --data-only
+Restore only the data, not the schema (data definitions).
+
+-c | --clean
+Clean (drop) database objects before recreating them.
+
+-C | --create
+Create the database before restoring into it. (When this option is used, the database named with `-d` is used only to issue the initial `CREATE DATABASE` command. All data is restored into the database name that appears in the archive.)
+
+-d *dbname* | --dbname=*dbname*
+Connect to this database and restore directly into this database. The default is to use the `PGDATABASE` environment variable setting, or the same name as the current system user.
+
+-e | --exit-on-error
+Exit if an error is encountered while sending SQL commands to the database. The default is to continue and to display a count of errors at the end of the restoration.
+
+-f *outfilename* | --file=*outfilename*
+Specify output file for generated script, or for the listing when used with `-l`. Default is the standard output.
+
+-F t |c | --format=tar | custom
+The format of the archive produced by [pg\_dump](pg_dump.html#topic1). It is not necessary to specify the format, since `pg_restore` will determine the format automatically. Format can be either `tar` or `custom`.
+
+-i | --ignore-version
+Ignore database version checks.
+
+-I *index* | --index=*index*
+Restore definition of named index only.
+
+-l | --list
+List the contents of the archive. The output of this operation can be used with the `-L` option to restrict and reorder the items that are restored.
+
+-L *list-file* | --use-list=*list-file*
+Restore elements in the *list-file* only, and in the order they appear in the file. Lines can be moved and may also be commented out by placing a `;` at the start of the line.
+
+-n *schema* | --schema=*schema*
+Restore only objects that are in the named schema. This can be combined with the `-t` option to restore just a specific table.
+
+-O | --no-owner
+Do not output commands to set ownership of objects to match the original database. By default, `pg_restore` issues `ALTER OWNER` or `SET SESSION AUTHORIZATION` statements to set ownership of created schema elements. These statements will fail unless the initial connection to the database is made by a superuser (or the same user that owns all of the objects in the script). With `-O`, any user name can be used for the initial connection, and this user will own all the created objects.
+
+-P '*function-name*(*argtype* \[, ...\])' | --function='*function-name*(*argtype* \[, ...\])'
+Restore the named function only. The function name must be enclosed in quotes. Be careful to spell the function name and arguments exactly as they appear in the dump file's table of contents (as shown by the `--list` option).
+
+-s | --schema-only
+Restore only the schema (data definitions), not the data (table contents). Sequence current values will not be restored, either. (Do not confuse this with the `--schema` option, which uses the word schema in a different meaning.)
+
+-S *username* | --superuser=*username*
+Specify the superuser user name to use when disabling triggers. This is only relevant if `--disable-triggers` is used.
+
+**Note:** HAWQ does not support user-defined triggers.
+
+-t *table* | --table=*table*
+Restore definition and/or data of named table only.
+
+-T *trigger* | --trigger=*trigger*
+Restore named trigger only.
+
+**Note:** HAWQ does not support user-defined triggers.
+
+-v | --verbose
+Specifies verbose mode.
+
+-x | --no-privileges | --no-acl
+Prevent restoration of access privileges (`GRANT/REVOKE` commands).
+
+--disable-triggers
+This option is only relevant when performing a data-only restore. It instructs `pg_restore` to execute commands to temporarily disable triggers on the target tables while the data is reloaded. Use this if you have triggers on the tables that you do not want to invoke during data reload. The commands emitted for `--disable-triggers` must be done as superuser. So, you should also specify a superuser name with `-S`, or preferably run `pg_restore` as a superuser.
+
+**Note:** HAWQ does not support user-defined triggers.
+
+--no-data-for-failed-tables
+By default, table data is restored even if the creation command for the table failed (e.g., because it already exists). With this option, data for such a table is skipped. This behavior is useful when the target database may already contain the desired table contents. Specifying this option prevents duplicate or obsolete data from being loaded. This option is effective only when restoring directly into a database, not when producing SQL script output.
+
+**Connection Options**
+
+-h *host* | --host *host*
+The host name of the machine on which the HAWQ master database server is running. If not specified, reads from the environment variable `PGHOST` or defaults to localhost.
+
+-p *port* | --port *port*
+The TCP port on which the HAWQ master database server is listening for connections. If not specified, reads from the environment variable `PGPORT` or defaults to 5432.
+
+-U *username* | --username *username*
+The database role name to connect as. If not specified, reads from the environment variable `PGUSER` or defaults to the current system role name.
+
+-W | --password
+Force a password prompt.
+
+-1 | --single-transaction
+Execute the restore as a single transaction. This ensures that either all the commands complete successfully, or no changes are applied.
+
+## Notes<a id="topic1__section6"></a>
+
+If your installation has any local additions to the `template1` database, be careful to load the output of `pg_restore` into a truly empty database; otherwise you are likely to get errors due to duplicate definitions of the added objects. To make an empty database without any local additions, copy from `template0` not `template1`, for example:
+
+``` pre
+CREATE DATABASE foo WITH TEMPLATE template0;
+```
+
+When restoring data to a pre-existing table and the option `--disable-triggers` is used, `pg_restore` emits commands to disable triggers on user tables before inserting the data then emits commands to re-enable them after the data has been inserted. If the restore is stopped in the middle, the system catalogs may be left in the wrong state.
+
+`pg_restore` will not restore large objects for a single table. If an archive contains large objects, then all large objects will be restored.
+
+See also the `pg_dump` documentation for details on limitations of `pg_dump`.
+
+Once restored, it is wise to run `ANALYZE` on each restored table so the query planner has useful statistics.
+
+When running `pg_restore`, a warning related to the `gp_enable_column_oriented_table` parameter might appear. If it does, disregard it.
+
+## Examples<a id="topic1__section7"></a>
+
+Assume we have dumped a database called `mydb` into a custom-format dump file:
+
+``` pre
+pg_dump -Fc mydb > db.dump
+```
+
+To drop the database and recreate it from the dump:
+
+``` pre
+dropdb mydb
+pg_restore -C -d template1 db.dump
+```
+
+To reload the dump into a new database called `newdb`. Notice there is no `-C`, we instead connect directly to the database to be restored into. Also note that we clone the new database from `template0` not `template1`, to ensure it is initially empty:
+
+``` pre
+createdb -T template0 newdb
+pg_restore -d newdb db.dump
+```
+
+To reorder database items, it is first necessary to dump the table of contents of the archive:
+
+``` pre
+pg_restore -l db.dump > db.list
+```
+
+The listing file consists of a header and one line for each item, for example,
+
+``` pre
+; Archive created at Fri Jul 28 22:28:36 2006
+; dbname: mydb
+; TOC Entries: 74
+; Compression: 0
+; Dump Version: 1.4-0
+; Format: CUSTOM
+;
+; Selected TOC Entries:
+;
+2; 145344 TABLE species postgres
+3; 145344 ACL species
+4; 145359 TABLE nt_header postgres
+5; 145359 ACL nt_header
+6; 145402 TABLE species_records postgres
+7; 145402 ACL species_records
+8; 145416 TABLE ss_old postgres
+9; 145416 ACL ss_old
+10; 145433 TABLE map_resolutions postgres
+11; 145433 ACL map_resolutions
+12; 145443 TABLE hs_old postgres
+13; 145443 ACL hs_old
+```
+
+Semicolons start a comment, and the numbers at the start of lines refer to the internal archive ID assigned to each item. Lines in the file can be commented out, deleted, and reordered. For example,
+
+``` pre
+10; 145433 TABLE map_resolutions postgres
+;2; 145344 TABLE species postgres
+;4; 145359 TABLE nt_header postgres
+6; 145402 TABLE species_records postgres
+;8; 145416 TABLE ss_old postgres
+```
+
+Could be used as input to `pg_restore` and would only restore items 10 and 6, in that order:
+
+`pg_restore -L db.list db.dump`
+
+## See Also<a id="topic1__section8"></a>
+
+[pg\_dump](pg_dump.html#topic1)
diff --git a/reference/cli/client_utilities/psql.html.md.erb b/reference/cli/client_utilities/psql.html.md.erb
new file mode 100644
index 0000000..c0d5954
--- /dev/null
+++ b/reference/cli/client_utilities/psql.html.md.erb
@@ -0,0 +1,690 @@
+---
+title: psql
+---
+
+Interactive command-line interface for HAWQ.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+psql [option ...] [dbname [username]]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`psql` is a terminal-based front-end to HAWQ. It enables you to type in queries interactively, issue them to HAWQ, and see the query results. Alternatively, input can be from a file. In addition, it provides a number of meta-commands and various shell-like features to facilitate writing scripts and automating a wide variety of tasks.
+
+**Note:** HAWQ queries timeout after a period of 600 seconds. For this reason, long-running queries may appear to hang in `plsql` until results are processed or until the timeout period expires.
+
+## Options<a id="topic1__section4"></a>
+
+-a | --echo-all
+Print all input lines to standard output as they are read. This is more useful for script processing rather than interactive mode.
+
+-A | --no-align
+Switches to unaligned output mode. (The default output mode is aligned.)
+
+-c '*command*' | --command '*command'*
+Specifies that `psql` is to execute the specified command string, and then exit. This is useful in shell scripts. *command* must be either a command string that is completely parseable by the server, or a single backslash command. Thus you cannot mix SQL and `psql` meta-commands with this option. To achieve that, you could pipe the string into `psql`, like this:
+
+``` pre
+echo '\x \\ SELECT * FROM foo;' | psql
+```
+
+(`\\` is the separator meta-command.)
+
+If the command string contains multiple SQL commands, they are processed in a single transaction, unless there are explicit `BEGIN/COMMIT` commands included in the string to divide it into multiple transactions. This is different from the behavior when the same string is fed to `psql`'s standard input.
+
+-d *dbname* | --dbname *dbname*
+Specifies the name of the database to connect to. This is equivalent to specifying dbname as the first non-option argument on the command line.
+
+If this parameter contains an equals sign, it is treated as a `conninfo` string; for example you can pass `'dbname=postgres user=username password=mypass'` as `dbname`.
+
+-e | --echo-queries
+Copy all SQL commands sent to the server to standard output as well.
+
+-E | --echo-hidden
+Echo the actual queries generated by `\d` and other backslash commands. You can use this to study `psql`'s internal operations.
+
+-f *filename* | --file *filename*
+Use a file as the source of commands instead of reading commands interactively. After the file is processed, `psql` terminates. If *filename* is `-` (hyphen), then standard input is read. Using this option is subtly different from writing `psql <`*filename*. In general, both will do what you expect, but using `-f` enables some nice features such as error messages with line numbers.
+
+-F *separator* | --field-separator *separator*
+Use the specified separator as the field separator for unaligned output.
+
+-H | --html
+Turn on HTML tabular output.
+
+-l | --list
+List all available databases, then exit. Other non-connection options are ignored.
+
+-L *filename* | --log-file *filename*
+Write all query output into the specified log file, in addition to the normal output destination.
+
+-o *filename* | --output *filename*
+Put all query output into the specified file.
+
+-P *assignment* | --pset *assignment*
+Allows you to specify printing options in the style of `\pset` on the command line. Note that here you have to separate name and value with an equal sign instead of a space. Thus to set the output format to LaTeX, you could write `-P format=latex`.
+
+-q | --quiet
+Specifies that `psql` should do its work quietly. By default, it prints welcome messages and various informational output. If this option is used, none of this happens. This is useful with the `-c` option.
+
+-R *separator* | --record-separator *separator*
+Use *separator* as the record separator for unaligned output.
+
+-s | --single-step
+Run in single-step mode. That means the user is prompted before each command is sent to the server, with the option to cancel execution as well. Use this to debug scripts.
+
+-S | --single-line
+Runs in single-line mode where a new line terminates an SQL command, as a semicolon does.
+
+-t | --tuples-only
+Turn off printing of column names and result row count footers, etc. This command is equivalent to `\pset tuples_only` and is provided for convenience.
+
+-T *table\_options* | --table-attr *table\_options*
+Allows you to specify options to be placed within the HTML table tag. See `\pset` for details.
+
+-v *assignment* | --set *assignment* | --variable *assignment*
+Perform a variable assignment, like the `\set` internal command. Note that you must separate name and value, if any, by an equal sign on the command line. To unset a variable, leave off the equal sign. To just set a variable without a value, use the equal sign but leave off the value. These assignments are done during a very early stage of start-up, so variables reserved for internal purposes might get overwritten later.
+
+-V | --version
+Print the `psql` version and exit.
+
+-x | --expanded
+Turn on the expanded table formatting mode.
+
+-X | --no-psqlrc
+Do not read the start-up file (neither the system-wide `psqlrc` file nor the user's `~/.psqlrc` file).
+
+-1 | --single-transaction
+When `psql` executes a script with the `-f` option, adding this option wraps `BEGIN/COMMIT` around the script to execute it as a single transaction. This ensures that either all the commands complete successfully, or no changes are applied.
+
+If the script itself uses `BEGIN`, `COMMIT`, or `ROLLBACK`, this option will not have the desired effects. Also, if the script contains any command that cannot be executed inside a transaction block, specifying this option will cause that command (and hence the whole transaction) to fail.
+
+-? | --help
+Show help about `psql` command line arguments, and exit.
+
+**Connection Options**
+
+-h *host* | --host *host*
+The host name of the machine on which the HAWQ master database server is running. If not specified, reads from the environment variable `PGHOST` or defaults to localhost.
+
+-p *port* | --port *port*
+The TCP port on which the HAWQ master database server is listening for connections. If not specified, reads from the environment variable `PGPORT` or defaults to 5432.
+
+-U *username* | --username *username*
+The database role name to connect as. If not specified, reads from the environment variable `PGUSER` or defaults to the current system role name.
+
+-W | --password
+Force a password prompt. `psql` should automatically prompt for a password whenever the server requests password authentication. However, currently password request detection is not totally reliable, hence this option to force a prompt. If no password prompt is issued and the server requires password authentication, the connection attempt will fail.
+
+-w --no-password
+Never issue a password prompt. If the server requires password authentication and a password is not available by other means such as a .pgpass file, the connection attempt will fail. This option can be useful in batch jobs and scripts where no user is present to enter a password.
+
+**Note:** This option remains set for the entire session, and so it affects uses of the meta-command `\connect` as well as the initial connection attempt.
+
+## Exit Status<a id="topic1__section6"></a>
+
+`psql` returns 0 to the shell if it finished normally, 1 if a fatal error of its own (out of memory, file not found) occurs, 2 if the connection to the server went bad and the session was not interactive, and 3 if an error occurred in a script and the variable `ON_ERROR_STOP` was set.
+
+## Usage<a id="topic1__section7"></a>
+
+**Connecting to a Database**
+
+`psql` is a client application for HAWQ. In order to connect to a database you need to know the name of your target database, the host name and port number of the HAWQ master server and what database user name you want to connect as. `psql` can be told about those parameters via command line options, namely `-d`, `-h`, `-p`, and `-U` respectively. If an argument is found that does not belong to any option it will be interpreted as the database name (or the user name, if the database name is already given). Not all these options are required; there are useful defaults. If you omit the host name, `psql` will connect via a UNIX-domain socket to a master server on the local host, or via TCP/IP to `localhost` on machines that do not have UNIX-domain sockets. The default master port number is 5432. If you use a different port for the master, you must specify the port. The default database user name is your UNIX user name, as is the default database name. Note that you cannot just connect to any database under any user name. Your database administrator should have informed you about your access rights.
+
+When the defaults are not right, you can save yourself some typing by setting any or all of the environment variables `PGAPPNAME`, `PGDATABASE`, `PGHOST`, `PGPORT`, and `PGUSER` to appropriate values.
+
+It is also convenient to have a `~/.pgpass` file to avoid regularly having to type in passwords. This file should reside in your home directory and contain lines of the following format:
+
+``` pre
+hostname:port:database:username:password
+```
+
+The permissions on `.pgpass` must disallow any access to world or group (for example: `chmod 0600 ~/.pgpass`). If the permissions are less strict than this, the file will be ignored. (The file permissions are not currently checked on Microsoft Windows clients, however.)
+
+If the connection could not be made for any reason (insufficient privileges, server is not running, etc.), `psql` will return an error and terminate.
+
+**Entering SQL Commands**
+
+In normal operation, `psql` provides a prompt with the name of the database to which `psql` is currently connected, followed by the string **=>** for a regular user or **=\#** for a superuser. For example:
+
+``` pre
+testdb=>
+testdb=#
+```
+
+At the prompt, the user may type in SQL commands. Ordinarily, input lines are sent to the server when a command-terminating semicolon is reached. An end of line does not terminate a command. Thus commands can be spread over several lines for clarity. If the command was sent and executed without error, the results of the command are displayed on the screen.
+
+## Meta-Commands<a id="topic1__section10"></a>
+
+Anything you enter in `psql` that begins with an unquoted backslash is a `psql` meta-command that is processed by `psql` itself. These commands help make `psql` more useful for administration or scripting. Meta-commands are more commonly called slash or backslash commands.
+
+The format of a `psql` command is the backslash, followed immediately by a command verb, then any arguments. The arguments are separated from the command verb and each other by any number of whitespace characters.
+
+To include whitespace into an argument you may quote it with a single quote. To include a single quote into such an argument, use two single quotes. Anything contained in single quotes is furthermore subject to C-like substitutions for `\n` (new line), `\t` (tab), `\digits` (octal), and `\xdigits` (hexadecimal).
+
+If an unquoted argument begins with a colon (`:`), it is taken as a `psql` variable and the value of the variable is used as the argument instead.
+
+Arguments that are enclosed in backquotes (`` ` ``) are taken as a command line that is passed to the shell. The output of the command (with any trailing newline removed) is taken as the argument value. The above escape sequences also apply in backquotes.
+
+Some commands take an SQL identifier (such as a table name) as argument. These arguments follow the syntax rules of SQL: Unquoted letters are forced to lowercase, while double quotes (`"`) protect letters from case conversion and allow incorporation of whitespace into the identifier. Within double quotes, paired double quotes reduce to a single double quote in the resulting name. For example, `FOO"BAR"BAZ` is interpreted as `fooBARbaz`, and `"A weird"" name"` becomes `A weird" name`.
+
+Parsing for arguments stops when another unquoted backslash occurs. This is taken as the beginning of a new meta-command. The special sequence `\\` (two backslashes) marks the end of arguments and continues parsing SQL commands, if any. That way SQL and `psql` commands can be freely mixed on a line. But in any case, the arguments of a meta-command cannot continue beyond the end of the line.
+
+The following meta-commands are defined:
+
+\\a
+If the current table output format is unaligned, it is switched to aligned. If it is not unaligned, it is set to unaligned. This command is kept for backwards compatibility. See `\pset` for a more general solution.
+
+\\cd \[*directory*\]
+Changes the current working directory. Without argument, changes to the current user's home directory. To print your current working directory, use `\!pwd`.
+
+\\C \[*title*\]
+Sets the title of any tables being printed as the result of a query or unset any such title. This command is equivalent to `\pset title`.
+
+\\c | \\connect \[*dbname* \[*username*\] \[*host*\] \[*port*\]\]
+Establishes a new connection. If the new connection is successfully made, the previous connection is closed. If any of dbname, username, host or port are omitted, the value of that parameter from the previous connection is used. If the connection attempt failed, the previous connection will only be kept if `psql` is in interactive mode. When executing a non-interactive script, processing will immediately stop with an error. This distinction was chosen as a user convenience against typos, and a safety mechanism that scripts are not accidentally acting on the wrong database.
+
+\\conninfo
+Displays information about the current connection including the database name, the user name, the type of connection (UNIX domain socket, `TCP/IP`, etc.), the host, and the port.
+
+\\copy {*table* \[(*column\_list*)\] | (*query*)} {from | to} {*filename* | stdin | stdout | pstdin | pstdout} \[with\] \[binary\] \[oids\] \[delimiter \[as\] '*character*'\] \[null \[as\] '*string*'\] \[csv \[header\] \[quote \[as\] 'character'\] \[escape \[as\] '*character*'\] \[force quote column\_list\] \[force not null column\_list\]\]
+Performs a frontend (client) copy. This is an operation that runs an SQL `COPY` command, but instead of the server reading or writing the specified file, `psql` reads or writes the file and routes the data between the server and the local file system. This means that file accessibility and privileges are those of the local user, not the server, and no SQL superuser privileges are required.
+
+The syntax of the command is similar to that of the SQL `COPY` command. Note that, because of this, special parsing rules apply to the `\copy` command. In particular, the variable substitution rules and backslash escapes do not apply.
+
+`\copy ... from stdin | to stdout` reads/writes based on the command input and output respectively. All rows are read from the same source that issued the command, continuing until `\.` is read or the stream reaches `EOF`. Output is sent to the same place as command output. To read/write from `psql`'s standard input or output, use `pstdin` or `pstdout`. This option is useful for populating tables in-line within a SQL script file.
+
+This operation is not as efficient as the SQL `COPY` command because all data must pass through the client/server connection.
+
+\\copyright
+Shows the copyright and distribution terms of PostgreSQL on which HAWQ is based.
+
+\\d \[*relation\_pattern*\] | \\d+ \[*relation\_pattern*\] | \\dS \[*relation\_pattern*\]
+For each relation (table, external table, view, index, or sequence) matching the relation pattern, show all columns, their types, the tablespace (if not the default) and any special attributes such as `NOT NULL` or defaults, if any. Associated indexes, constraints, rules, and triggers are also shown, as is the view definition if the relation is a view.
+
+- The command form `\d+` is identical, except that more information is displayed: any comments associated with the columns of the table are shown, as is the presence of OIDs in the table.
+- The command form `\dS` is identical, except that system information is displayed as well as user information.For example, `\dt` displays user tables, but not system tables; `\dtS` displays both user and system tables.Both these commands can take the `+` parameter to display additional information, as in `\dt+` and `\dtS+`.
+
+ If `\d` is used without a pattern argument, it is equivalent to `\dtvs` which will show a list of all tables, views, and sequences.
+
+\\da \[*aggregate\_pattern*\]
+Lists all available aggregate functions, together with the data types they operate on. If a pattern is specified, only aggregates whose names match the pattern are shown.
+
+\\db \[*tablespace\_pattern*\] | \\db+ \[*tablespace\_pattern*\]
+Lists all available tablespaces and their corresponding filespace locations. If pattern is specified, only tablespaces whose names match the pattern are shown. If + is appended to the command name, each object is listed with its associated permissions.
+
+\\dc \[*conversion\_pattern*\]
+Lists all available conversions between character-set encodings. If pattern is specified, only conversions whose names match the pattern are listed.
+
+\\dC
+Lists all available type casts.
+
+\\dd \[*object\_pattern*\]
+Lists all available objects. If pattern is specified, only matching objects are shown.
+
+\\dD \[*domain\_pattern*\]
+Lists all available domains. If pattern is specified, only matching domains are shown.
+
+\\df \[*function\_pattern*\] | \\df+ \[function\_pattern \]
+Lists available functions, together with their argument and return types. If pattern is specified, only functions whose names match the pattern are shown. If the form `\df+` is used, additional information about each function, including language and description, is shown. To reduce clutter, `\df `does not show data type I/O functions. This is implemented by ignoring functions that accept or return type `cstring`.
+
+\\dg \[*role\_pattern*\]
+Lists all database roles. If pattern is specified, only those roles whose names match the pattern are listed.
+
+\\distPvxS \[index | sequence | table | parent table | view | external\_table | system\_object\]
+This is not the actual command name: the letters `i`, `s`, `t`, `P`, `v`, `x`, `S` stand for index, sequence, table, parent table, view, external table, and system table, respectively. You can specify any or all of these letters, in any order, to obtain a listing of all the matching objects. The letter `S` restricts the listing to system objects; without `S`, only non-system objects are shown. If + is appended to the command name, each object is listed with its associated description, if any. If a pattern is specified, only objects whose names match the pattern are listed.
+
+\\dl
+This is an alias for `\lo_list`, which shows a list of large objects.
+
+\\dn \[*schema\_pattern*\] | \\dn+ \[*schema\_pattern*\]
+Lists all available schemas (namespaces). If pattern is specified, only schemas whose names match the pattern are listed. Non-local temporary schemas are suppressed. If `+` is appended to the command name, each object is listed with its associated permissions and description, if any.
+
+\\do \[*operator\_pattern*\]
+Lists available operators with their operand and return types. If pattern is specified, only operators whose names match the pattern are listed.
+
+\\dp \[*relation\_pattern\_to\_show\_privileges*\]
+Produces a list of all available tables, views and sequences with their associated access privileges. If pattern is specified, only tables, views and sequences whose names match the pattern are listed. The `GRANT` and `REVOKE` commands are used to set access privileges.
+
+\\dT \[*datatype\_pattern*\] | \\dT+ \[*datatype\_pattern*\]
+Lists all data types or only those that match pattern. The command form `\dT+` shows extra information.
+
+\\du \[*role\_pattern*\]
+Lists all database roles, or only those that match pattern.
+
+\\e | \\edit \[*filename*\]
+If a file name is specified, the file is edited; after the editor exits, its content is copied back to the query buffer. If no argument is given, the current query buffer is copied to a temporary file which is then edited in the same fashion. The new query buffer is then re-parsed according to the normal rules of `psql`, where the whole buffer is treated as a single line. (Thus you cannot make scripts this way. Use `\i` for that.) This means also that if the query ends with (or rather contains) a semicolon, it is immediately executed. In other cases it will merely wait in the query buffer.
+
+`psql` searches the environment variables `PSQL_EDITOR`, `EDITOR`, and `VISUAL` (in that order) for an editor to use. If all of them are unset, `vi` is used on UNIX systems, `notepad.exe` on Windows systems.
+
+\\echotext \[ ... \]
+Prints the arguments to the standard output, separated by one space and followed by a newline. This can be useful to intersperse information in the output of scripts.
+
+If you use the `\o` command to redirect your query output you may wish to use` 'echo` instead of this command.
+
+\\encoding \[*encoding*\]
+Sets the client character set encoding. Without an argument, this command shows the current encoding.
+
+\\f \[*field\_separator\_string*\]
+Sets the field separator for unaligned query output. The default is the vertical bar (`|`). See also `\pset` for a generic way of setting output options.
+
+\\g \[{*filename* | |*command* }\]
+Sends the current query input buffer to the server and optionally stores the query's output in a file or pipes the output into a separate UNIX shell executing command. A bare `\g` is virtually equivalent to a semicolon. A `\g` with argument is a one-shot alternative to the `\o` command.
+
+\\h | \\help \[*sql\_command*\]
+Gives syntax help on the specified SQL command. If a command is not specified, then `psql` will list all the commands for which syntax help is available. Use an asterisk (\*) to show syntax help on all SQL commands. To simplify typing, commands that consists of several words do not have to be quoted.
+
+\\H
+Turns on HTML query output format. If the HTML format is already on, it is switched back to the default aligned text format. This command is for compatibility and convenience, but see `\pset` about setting other output options.
+
+\\i *input\_filename*
+Reads input from a file and executes it as though it had been typed on the keyboard. If you want to see the lines on the screen as they are read you must set the variable `ECHO` to all.
+
+\\l | \\list | \\l+ | \\list+
+List the names, owners, and character set encodings of all the databases in the server. If `+` is appended to the command name, database descriptions are also displayed.
+
+\\lo\_export *loid* *filename*
+Reads the large object with OID *loid* from the database and writes it to *filename*. Note that this is subtly different from the server function `lo_export`, which acts with the permissions of the user that the database server runs as and on the server's file system. Use `\lo_list` to find out the large object's OID.
+
+\\lo\_import *large\_object\_filename* \[*comment*\]
+Stores the file into a large object. Optionally, it associates the given comment with the object. Example:
+
+``` pre
+mydb=> \lo_import '/home/gpadmin/pictures/photo.xcf' 'a
+picture of me'
+lo_import 152801
+```
+
+The response indicates that the large object received object ID 152801 which one ought to remember if one wants to access the object ever again. For that reason, you should always associate a human-readable comment with every object. Those can then be seen with the `\lo_list` command. Note that this command is subtly different from the server-side `lo_import` because it acts as the local user on the local file system, rather than the server's user and file system.
+
+\\lo\_list
+Shows a list of all large objects currently stored in the database, along with any comments provided for them.
+
+\\lo\_unlink *largeobject\_oid*
+Deletes the large object of the specified OID from the database. Use `\lo_list` to find out the large object's OID.
+
+\\o \[ {*query\_result\_filename* | |*command*} \]
+Saves future query results to a file or pipes them into a UNIX shell command. If no arguments are specified, the query output will be reset to the standard output. Query results include all tables, command responses, and notices obtained from the database server, as well as output of various backslash commands that query the database (such as `\d`), but not error messages. To intersperse text output in between query results, use `'echo`.
+
+\\p
+Print the current query buffer to the standard output.
+
+\\password \[*username*\]
+Changes the password of the specified user (by default, the current user). This command prompts for the new password, encrypts it, and sends it to the server as an `ALTER ROLE` command. This makes sure that the new password does not appear in cleartext in the command history, the server log, or elsewhere.
+
+\\prompt \[ *text* \] *name*
+Prompts the user to set a variable *name*. Optionally, you can specify a prompt. Enclose prompts longer than one word in single quotes.
+
+By default, \\prompt uses the terminal for input and output. However, use the -f command line switch to specify standard input and standard output.
+
+\\pset *print\_option* \[*value*\]
+This command sets options affecting the output of query result tables. *print\_option* describes which option is to be set. Adjustable printing options are:
+
+- **`format`** – Sets the output format to one of **u**`naligned`, **a**`ligned`, **h**`tml`, **l**`atex`, **t**`roff-ms`, or **w**`rapped`. First letter abbreviations are allowed. Unaligned writes all columns of a row on a line, separated by the currently active field separator. This is intended to create output that might be intended to be read in by other programs. Aligned mode is the standard, human-readable, nicely formatted text output that is default. The HTML and LaTeX modes put out tables that are intended to be included in documents using the respective mark-up language. They are not complete documents! (This might not be so dramatic in HTML, but in LaTeX you must have a complete document wrapper.)
+
+ The wrapped option sets the output format like the `aligned` parameter , but wraps wide data values across lines to make the output fit in the target column width. The target width is set with the `columns` option. To specify the column width and select the wrapped format, use two \\pset commands; for example, to set the with to 72 columns and specify wrapped format, use the commands `\pset columns 72` and then `\pset format wrapped`.
+
+ **Note:** Since `psql` does not attempt to wrap column header titles, the wrapped format behaves the same as aligned if the total width needed for column headers exceeds the target.
+
+- **`border`** – The second argument must be a number. In general, the higher the number the more borders and lines the tables will have, but this depends on the particular format. In HTML mode, this will translate directly into the `border=...` attribute, in the others only values `0` (no border), `1` (internal dividing lines), and `2` (table frame) make sense.
+- **`columns`** – Sets the target width for the `wrapped` format, and also the width limit for determining whether output is wide enough to require the pager. The default is *zero*. Zero causes the target width to be controlled by the environment variable `COLUMNS`, or the detected screen width if `COLUMNS` is not set. In addition, if `columns` is zero then the wrapped format affects screen output only. If columns is nonzero then file and pipe output is wrapped to that width as well.
+
+ After setting the target width, use the command `\pset format wrapped` to enable the wrapped format.
+
+- **`expanded`** | **`x)`** – Toggles between regular and expanded format. When expanded format is enabled, query results are displayed in two columns, with the column name on the left and the data on the right. This mode is useful if the data would not fit on the screen in the normal horizontal mode. Expanded mode is supported by all four output formats.
+- **`linestyle`** \[**`unicode`** | **`ascii`** | **`old-ascii`**\] – Sets the border line drawing style to one of unicode, ascii, or old-ascii. Unique abbreviations, including one letter, are allowed for the three styles. The default setting is `ascii`. This option only affects the `aligned` and `wrapped` output formats.
+
+ **`ascii`** – uses plain ASCII characters. Newlines in data are shown using a + symbol in the right-hand margin. When the wrapped format wraps data from one line to the next without a newline character, a dot (.) is shown in the right-hand margin of the first line, and again in the left-hand margin of the following line.
+
+ **`old-ascii`** – style uses plain ASCII characters, using the formatting style used in PostgreSQL 8.4 and earlier. Newlines in data are shown using a : symbol in place of the left-hand column separator. When the data is wrapped from one line to the next without a newline character, a ; symbol is used in place of the left-hand column separator.
+
+ **`unicode`** – style uses Unicode box-drawing characters. Newlines in data are shown using a carriage return symbol in the right-hand margin. When the data is wrapped from one line to the next without a newline character, an ellipsis symbol is shown in the right-hand margin of the first line, and again in the left-hand margin of the following line.
+
+ When the `border` setting is greater than zero, this option also determines the characters with which the border lines are drawn. Plain ASCII characters work everywhere, but Unicode characters look nicer on displays that recognize them.
+
+- **`null 'string'`** – The second argument is a string to print whenever a column is null. The default is not to print anything, which can easily be mistaken for an empty string. For example, the command `\pset``null '(empty)' `displays *(empty)* in null columns.
+- **`fieldsep`** – Specifies the field separator to be used in unaligned output mode. That way one can create, for example, tab- or comma-separated output, which other programs might prefer. To set a tab as field separator, type `\pset fieldsep '\t'`. The default field separator is `'|'` (a vertical bar).
+- **`footer`** – Toggles the display of the default footer (*x* rows).
+- **`numericlocale`** – Toggles the display of a locale-aware character to separate groups of digits to the left of the decimal marker. It also enables a locale-aware decimal marker.
+- **`recordsep`** – Specifies the record (line) separator to use in unaligned output mode. The default is a newline character.
+- **`title`** \[*text*\] – Sets the table title for any subsequently printed tables. This can be used to give your output descriptive tags. If no argument is given, the title is unset.
+- **`tableattr`** | **`T`** \[*text*\] – Allows you to specify any attributes to be placed inside the HTML table tag. This could for example be `cellpadding` or `bgcolor`. Note that you probably don't want to specify border here, as that is already taken care of by `\pset border`.
+- **`tuples_only`** | **`t `** \[*novalue* | *on* | *off*\] – The `\pset tuples_only` command by itselt toggles between tuples only and full display. The values *on* and *off* set the tuples display, regardless of the current setting. Full display may show extra information such as column headers, titles, and various footers. In tuples only mode, only actual table data is shown The `\t` command is equivalent to `\pset``tuples_only` and is provided for convenience.
+- **`pager`** – Controls the use of a pager for query and `psql` help output. When `on`, if the environment variable `PAGER` is set, the output is piped to the specified program. Otherwise a platform-dependent default (such as `more`) is used. When `off`, the pager is not used. When `on`, the pager is used only when appropriate. Pager can also be set to `always`, which causes the pager to be always used.
+
+\\q
+Quits the `psql` program.
+
+\\qechotext \[ ... \]
+This command is identical to `\echo` except that the output will be written to the query output channel, as set by `\o`.
+
+\\r
+Resets (clears) the query buffer.
+
+\\s \[*history\_filename*\]
+Print or save the command line history to *filename*. If *filename* is omitted, the history is written to the standard output.
+
+\\set \[*name* \[*value* \[ ... \]\]\]
+Sets the internal variable *name* to *value* or, if more than one value is given, to the concatenation of all of them. If no second argument is given, the variable is just set with no value. To unset a variable, use the `\unset` command.
+
+Valid variable names can contain characters, digits, and underscores. See "Variables" in [Advanced Features](#topic1__section12). Variable names are case-sensitive.
+
+Although you are welcome to set any variable to anything you want, `psql` treats several variables as special. They are documented in the topic about variables.
+
+This command is totally separate from the SQL command `SET`.
+
+\\t \[novalue | on | off\]
+The `\t` command by itself toggles a display of output column name headings and row count footer. The values `on` and `off` set the tuples display, regardless of the current setting.This command is equivalent to `\pset tuples_only` and is provided for convenience.
+
+\\T *table\_options*
+Allows you to specify attributes to be placed within the table tag in HTML tabular output mode.
+
+\\timing \[novalue | on | off\]
+The `\timing` command by itself toggles a display of how long each SQL statement takes, in milliseconds. The values `on` and `off` set the time display, regardless of the current setting.
+
+\\w {*filename* | |*command*}
+Outputs the current query buffer to a file or pipes it to a UNIX command.
+
+\\x
+Toggles expanded table formatting mode.
+
+\\z \[*relation\_to\_show\_privileges*\]
+Produces a list of all available tables, views and sequences with their associated access privileges. If a pattern is specified, only tables, views and sequences whose names match the pattern are listed. This is an alias for `\dp`.
+
+\\! \[*command*\]
+Escapes to a separate UNIX shell or executes the UNIX command. The arguments are not further interpreted, the shell will see them as is.
+
+\\?
+Shows help information about the `psql` backslash commands.
+
+## Patterns<a id="topic1__section11"></a>
+
+The various `\d` commands accept a pattern parameter to specify the object name(s) to be displayed. In the simplest case, a pattern is just the exact name of the object. The characters within a pattern are normally folded to lower case, just as in SQL names; for example, `\dt FOO` will display the table named `foo`. As in SQL names, placing double quotes around a pattern stops folding to lower case. Should you need to include an actual double quote character in a pattern, write it as a pair of double quotes within a double-quote sequence; again this is in accord with the rules for SQL quoted identifiers. For example, `\dt "FOO""BAR"` will display the table named `FOO"BAR` (not `foo"bar`). Unlike the normal rules for SQL names, you can put double quotes around just part of a pattern, for instance `\dt FOO"FOO"BAR` will display the table named `fooFOObar`.
+
+Within a pattern, `*` matches any sequence of characters (including no characters) and `?` matches any single character. (This notation is comparable to UNIX shell file name patterns.) For example, `\dt int*` displays all tables whose names begin with `int`. But within double quotes, `*` and `?` lose these special meanings and are just matched literally.
+
+A pattern that contains a dot (`.`) is interpreted as a schema name pattern followed by an object name pattern. For example, `\dt foo*.bar*` displays all tables whose table name starts with `bar` that are in schemas whose schema name starts with `foo`. When no dot appears, then the pattern matches only objects that are visible in the current schema search path. Again, a dot within double quotes loses its special meaning and is matched literally.
+
+Advanced users can use regular-expression notations. All regular expression special characters work as specified in the [PostgreSQL documentation on regular expressions](http://www.postgresql.org/docs/8.2/static/functions-matching.html#FUNCTIONS-POSIX-REGEXP), except for `.` which is taken as a separator as mentioned above, `*` which is translated to the regular-expression notation `.*`, and `?` which is translated to `..` You can emulate these pattern characters at need by writing `?` for `.,``(R+|)` for `R*`, or `(R|)` for `R?`. Remember that the pattern must match the whole name, unlike the usual interpretation of regular expressions; write `*` at the beginning and/or end if you don't wish the pattern to be anchored. Note that within double quotes, all regular expression special characters lose their special meanings and are matched literally. Also, the regular expression special characters are matched literally in operator name patterns (such as the argument of `\do`).
+
+Whenever the pattern parameter is omitted completely, the `\d` commands display all objects that are visible in the current schema search path – this is equivalent to using the pattern `*.` To see all objects in the database, use the pattern `*.*.`
+
+## Advanced Features<a id="topic1__section12"></a>
+
+**Variables**
+
+`psql` provides variable substitution features similar to common UNIX command shells. Variables are simply name/value pairs, where the value can be any string of any length. To set variables, use the `psql` meta-command `\set`:
+
+``` pre
+testdb=> \set foo bar
+```
+
+sets the variable `foo` to the value `bar`. To retrieve the content of the variable, precede the name with a colon and use it as the argument of any slash command:
+
+``` pre
+testdb=> \echo :foo
+bar
+```
+
+**Note:** The arguments of `\set` are subject to the same substitution rules as with other commands. Thus you can construct interesting references such as `\set :foo 'something'` and get 'soft links' or 'variable variables' of Perl or PHP fame, respectively. Unfortunately, there is no way to do anything useful with these constructs. On the other hand, `\set bar :foo` is a perfectly valid way to copy a variable.
+
+If you call `\set` without a second argument, the variable is set, with an empty string as *value*. To unset (or delete) a variable, use the command `\unset`.
+
+`psql`'s internal variable names can consist of letters, numbers, and underscores in any order and any number of them. A number of these variables are treated specially by `psql`. They indicate certain option settings that can be changed at run time by altering the value of the variable or represent some state of the application. Although you can use these variables for any other purpose, this is not recommended, as the program behavior might behave unexpectedly. By convention, all specially treated variables consist of all upper-case letters (and possibly numbers and underscores). To ensure maximum compatibility in the future, avoid using such variable names for your own purposes. A list of all specially treated variables are as follows:
+
+AUTOCOMMIT
+When on (the default), each SQL command is automatically committed upon successful completion. To postpone commit in this mode, you must enter a `BEGIN` or `START TRANSACTION` SQL command. When off or unset, SQL commands are not committed until you explicitly issue `COMMIT` or `END`. The autocommit-on mode works by issuing an implicit `BEGIN` for you, just before any command that is not already in a transaction block and is not itself a `BEGIN` or other transaction-control command, nor a command that cannot be executed inside a transaction block (such as `VACUUM`).
+
+In autocommit-off mode, you must explicitly abandon any failed transaction by entering `ABORT` or `ROLLBACK`. Also keep in mind that if you exit the session without committing, your work will be lost.
+
+The autocommit-on mode is PostgreSQL's traditional behavior, but autocommit-off is closer to the SQL spec. If you prefer autocommit-off, you may wish to set it in your `~/.psqlrc` file.
+
+DBNAME
+The name of the database you are currently connected to. This is set every time you connect to a database (including program start-up), but can be unset.
+
+ECHO
+If set to all, all lines entered from the keyboard or from a script are written to the standard output before they are parsed or executed. To select this behavior on program start-up, use the switch `-a`. If set to queries, `psql` merely prints all queries as they are sent to the server. The switch for this is `-e`.
+
+ECHO\_HIDDEN
+When this variable is set and a backslash command queries the database, the query is first shown. This way you can study the HAWQ internals and provide similar functionality in your own programs. (To select this behavior on program start-up, use the switch `-E`.) If you set the variable to the value `noexec`, the queries are just shown but are not actually sent to the server and executed.
+
+ENCODING
+The current client character set encoding.
+
+FETCH\_COUNT
+If this variable is set to an integer value > 0, the results of `SELECT` queries are fetched and displayed in groups of that many rows, rather than the default behavior of collecting the entire result set before display. Therefore only a limited amount of memory is used, regardless of the size of the result set. Settings of 100 to 1000 are commonly used when enabling this feature. Keep in mind that when using this feature, a query may fail after having already displayed some rows.
+
+Although you can use any output format with this feature, the default aligned format tends to look bad because each group of `FETCH_COUNT` rows will be formatted separately, leading to varying column widths across the row groups. The other output formats work better.
+
+HISTCONTROL
+If this variable is set to `ignorespace`, lines which begin with a space are not entered into the history list. If set to a value of `ignoredups`, lines matching the previous history line are not entered. A value of `ignoreboth` combines the two options. If unset, or if set to any other value than those above, all lines read in interactive mode are saved on the history list.
+
+HISTFILE
+The file name that will be used to store the history list. The default value is `~/.psql_history`. For example, putting
+
+``` pre
+\set HISTFILE ~/.psql_history- :DBNAME
+```
+
+in `~/.psqlrc` will cause `psql` to maintain a separate history for each database.
+
+HISTSIZE
+The number of commands to store in the command history. The default value is 500.
+
+HOST
+The database server host you are currently connected to. This is set every time you connect to a database (including program start-up), but can be unset.
+
+IGNOREEOF
+If unset, sending an `EOF` character (usually `CTRL+D`) to an interactive session of `psql` will terminate the application. If set to a numeric value, that many `EOF` characters are ignored before the application terminates. If the variable is set but has no numeric value, the default is `10`.
+
+LASTOID
+The value of the last affected OID, as returned from an `INSERT` or `lo_insert` command. This variable is only guaranteed to be valid until after the result of the next SQL command has been displayed.
+
+ON\_ERROR\_ROLLBACK
+When on, if a statement in a transaction block generates an error, the error is ignored and the transaction continues. When interactive, such errors are only ignored in interactive sessions, and not when reading script files. When off (the default), a statement in a transaction block that generates an error aborts the entire transaction. The on\_error\_rollback-on mode works by issuing an implicit `SAVEPOINT` for you, just before each command that is in a transaction block, and rolls back to the savepoint on error.
+
+ON\_ERROR\_STOP
+By default, if non-interactive scripts encounter an error, such as a malformed SQL command or internal meta-command, processing continues. This has been the traditional behavior of `psql` but it is sometimes not desirable. If this variable is set, script processing will immediately terminate. If the script was called from another script it will terminate in the same fashion. If the outermost script was not called from an interactive `psql` session but rather using the `-f` option, `psql` will return error code 3, to distinguish this case from fatal error conditions (error code 1).
+
+PORT
+The database server port to which you are currently connected. This is set every time you connect to a database (including program start-up), but can be unset.
+
+PROMPT1
+PROMPT2
+PROMPT3
+These specify what the prompts `psql` issues should look like. See "Prompting".
+
+QUIET
+This variable is equivalent to the command line option `-q`. It is not very useful in interactive mode.
+
+SINGLELINE
+This variable is equivalent to the command line option `-S`.
+
+SINGLESTEP
+This variable is equivalent to the command line option `-s`.
+
+USER
+The database user you are currently connected as. This is set every time you connect to a database (including program start-up), but can be unset.
+
+VERBOSITY
+This variable can be set to the values `default`, `verbose`, or `terse` to control the verbosity of error reports.
+
+**SQL Interpolation**
+An additional useful feature of `psql` variables is that you can substitute (interpolate) them into regular SQL statements. The syntax for this is again to prepend the variable name with a colon (`:`).
+
+``` pre
+testdb=> \set foo 'my_table'
+testdb=> SELECT * FROM :foo;
+```
+
+would then query the table `my_table`. The value of the variable is copied literally, so it can even contain unbalanced quotes or backslash commands. You must make sure that it makes sense where you put it. Variable interpolation will not be performed into quoted SQL entities.
+
+A popular application of this facility is to refer to the last inserted OID in subsequent statements to build a foreign key scenario. Another possible use of this mechanism is to copy the contents of a file into a table column. First load the file into a variable and then proceed as above.
+
+``` pre
+testdb=> \set content '''' `cat my_file.txt` ''''
+testdb=> INSERT INTO my_table VALUES (:content);
+```
+
+One problem with this approach is that `my_file.txt` might contain single quotes. These need to be escaped so that they don't cause a syntax error when the second line is processed. This could be done with the program `sed`:
+
+``` pre
+testdb=> \set content '''' `sed -e "s/'/''/g" < my_file.txt`
+''''
+```
+
+If you are using non-standard-conforming strings then you'll also need to double backslashes. This is a bit tricky:
+
+``` pre
+testdb=> \set content '''' `sed -e "s/'/''/g" -e
+'s/\\/\\\\/g' < my_file.txt` ''''
+```
+
+Note the use of different shell quoting conventions so that neither the single quote marks nor the backslashes are special to the shell. Backslashes are still special to `sed`, however, so we need to double them.
+
+Since colons may legally appear in SQL commands, the following rule applies: the character sequence `":name"` is not changed unless `"name"` is the name of a variable that is currently set. In any case you can escape a colon with a backslash to protect it from substitution. (The colon syntax for variables is standard SQL for embedded query languages, such as ECPG. The colon syntax for array slices and type casts are HAWQ extensions, hence the conflict.)
+
+**Prompting**
+
+The prompts `psql` issues can be customized to your preference. The three variables `PROMPT1`, `PROMPT2`, and `PROMPT3` contain strings and special escape sequences that describe the appearance of the prompt. Prompt 1 is the normal prompt that is issued when `psql` requests a new command. Prompt 2 is issued when more input is expected during command input because the command was not terminated with a semicolon or a quote was not closed. Prompt 3 is issued when you run an SQL `COPY` command and you are expected to type in the row values on the terminal.
+
+The value of the selected prompt variable is printed literally, except where a percent sign (`%`) is encountered. Depending on the next character, certain other text is substituted instead. Defined substitutions are:
+
+%M
+The full host name (with domain name) of the database server, or `[local]` if the connection is over a UNIX domain socket, or `[local:/dir/name]`, if the UNIX domain socket is not at the compiled in default location.
+
+%m
+The host name of the database server, truncated at the first dot, or `[local]` if the connection is over a UNIX domain socket.
+
+%>
+The port number at which the database server is listening.
+
+%n
+The database session user name. (The expansion of this value might change during a database session as the result of the command `SET SESSION AUTHORIZATION`.)
+
+%/
+The name of the current database.
+
+%~
+Like `%/`, but the output is `~` (tilde) if the database is your default database.
+
+%\#
+If the session user is a database superuser, then a **\#**, otherwise a **>**. (The expansion of this value might change during a database session as the result of the command `SET SESSION AUTHORIZATION`.)
+
+%R
+In prompt 1 normally **=**, but **^** if in single-line mode, and **!** if the session is disconnected from the database (which can happen if `\connect` fails). In prompt 2 the sequence is replaced by **-**, **\***, a single quote, a double quote, or a dollar sign, depending on whether `psql` expects more input because the command wasn't terminated yet, because you are inside a `/* ... */` comment, or because you are inside a quoted or dollar-escaped string. In prompt 3 the sequence doesn't produce anything.
+
+%x
+Transaction status: an empty string when not in a transaction block, or **\*** when in a transaction block, or **!** when in a failed transaction block, or **?** when the transaction state is indeterminate (for example, because there is no connection).
+
+%digits
+The character with the indicated octal code is substituted.
+
+%:name:
+The value of the `psql` variable name. See "Variables" in [Advanced Features](#topic1__section12) for details.
+
+%\`command\`
+The output of command, similar to ordinary back-tick substitution.
+
+%\[ ... %\]
+Prompts may contain terminal control characters which, for example, change the color, background, or style of the prompt text, or change the title of the terminal window. In order for line editing to work properly, these non-printing control characters must be designated as invisible by surrounding them with `%[` and `%]`. Multiple pairs of these may occur within the prompt. For example,
+
+``` pre
+testdb=> \set PROMPT1 '%[%033[1;33;40m%]%n@%/%R%[%033[0m%]%#'
+```
+
+results in a boldfaced (`1;`) yellow-on-black (`33;40`) prompt on VT100-compatible, color-capable terminals. To insert a percent sign into your prompt, write `%%`. The default prompts are `'%/%R%# '` for prompts 1 and 2, and `'>> '` for prompt 3.
+
+**Command-Line Editing**
+
+`psql` supports the NetBSD libedit library for convenient line editing and retrieval. The command history is automatically saved when `psql` exits and is reloaded when `psql` starts up. Tab-completion is also supported, although the completion logic makes no claim to be an SQL parser. If for some reason you do not like the tab completion, you can turn it off by putting this in a file named `.inputrc` in your home directory:
+
+``` pre
+$if psql
+set disable-completion on
+$endif
+```
+
+## Environment<a id="topic1__section17"></a>
+
+PAGER
+If the query results do not fit on the screen, they are piped through this command. Typical values are `more` or `less`. The default is platform-dependent. The use of the pager can be disabled by using the `\pset` command.
+
+PGDATABASE
+PGHOST
+PGPORT
+PGUSER
+Default connection parameters.
+
+PSQL\_EDITOR
+EDITOR
+VISUAL
+Editor used by the `\e` command. The variables are examined in the order listed; the first that is set is used.
+
+SHELL
+Command executed by the `\!` command.
+
+TMPDIR
+Directory for storing temporary files. The default is `/tmp`.
+
+## Files<a id="topic1__section18"></a>
+
+Before starting up, `psql` attempts to read and execute commands from the user's `~/.psqlrc` file.
+
+The command-line history is stored in the file `~/.psql_history`.
+
+## Notes<a id="topic1__section19"></a>
+
+`psql` only works smoothly with servers of the same version. That does not mean other combinations will fail outright, but subtle and not-so-subtle problems might come up. Backslash commands are particularly likely to fail if the server is of a different version.
+
+## Notes for Windows users<a id="topic1__section20"></a>
+
+`psql` is built as a console application. Since the Windows console windows use a different encoding than the rest of the system, you must take special care when using 8-bit characters within `psql`. If `psql` detects a problematic console code page, it will warn you at startup. To change the console code page, two things are necessary:
+
+Set the code page by entering:
+
+``` pre
+cmd.exe /c chcp 1252
+```
+
+`1252` is a character encoding of the Latin alphabet, used by Microsoft Windows for English and some other Western languages. If you are using Cygwin, you can put this command in `/etc/profile`.
+
+Set the console font to Lucida Console, because the raster font does not work with the ANSI code page.
+
+## Examples<a id="topic1__section21"></a>
+
+Start `psql` in interactive mode:
+
+``` pre
+psql -p 54321 -U sally mydatabase
+```
+
+In `psql` interactive mode, spread a command over several lines of input. Notice the changing prompt:
+
+``` pre
+testdb=> CREATE TABLE my_table (
+testdb(> first integer not null default 0,
+testdb(> second text)
+testdb-> ;
+CREATE TABLE
+```
+
+Look at the table definition:
+
+``` pre
+testdb=> \d my_table
+ Table "my_table"
+ Attribute | Type | Modifier
+-----------+---------+--------------------
+ first | integer | not null default 0
+ second | text |
+```
+
+Run `psql` in non-interactive mode by passing in a file containing SQL commands:
+
+``` pre
+psql -f /home/gpadmin/test/myscript.sql
+```
diff --git a/reference/cli/client_utilities/vacuumdb.html.md.erb b/reference/cli/client_utilities/vacuumdb.html.md.erb
new file mode 100644
index 0000000..e063799
--- /dev/null
+++ b/reference/cli/client_utilities/vacuumdb.html.md.erb
@@ -0,0 +1,94 @@
+---
+title: vacuumdb
+---
+
+Garbage-collects and analyzes a database.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+vacuumdb [connection-option...] [--full | -f] [-F] [--verbose | -v]
+ [--analyze | -z] [--table | -t table [( column [,...] )] ] [dbname]
+
+vacuumdb [connection-options...] [--all | -a] [--full | -f] [-F]
+ [--verbose | -v] [--analyze | -z]
+
+vacuumdb --help
+
+vacuumdb --version
+```
+
+## Description<a id="topic1__section3"></a>
+
+`vacuumdb` is a utility for cleaning a PostgreSQL database. vacuumdb will also generate internal statistics used by the PostgreSQL query optimizer.
+
+`vacuumdb` is a wrapper around the SQL command `VACUUM`. There is no effective difference between vacuuming databases via this utility and via other methods for accessing the server.
+
+## Options<a id="topic1__section4"></a>
+
+-a | --all
+Vacuums all databases.
+
+\[-d\] *dbname* | \[--dbname\] *dbname*
+The name of the database to vacuum. If this is not specified and `-all` is not used, the database name is read from the environment variable `PGDATABASE`. If that is not set, the user name specified for the connection is used.
+
+-f | --full
+Selects a full vacuum, which may reclaim more space, but takes much longer and exclusively locks the table.
+
+**Warning:** A `VACUUM FULL` is not recommended in HAWQ.
+
+-F | --freeze
+Freeze row transaction information.
+
+-q | --quiet
+Do not display a response.
+
+-t *table* \[(*column*)\] | --table *table* \[(*column*)\]
+Clean or analyze this table only. Column names may be specified only in conjunction with the `--analyze` option. If you specify columns, you probably have to escape the parentheses from the shell.
+
+-v | --verbose
+Print detailed information during processing.
+
+-z | --analyze
+Collect statistics for use by the query planner.
+
+**Connection Options**
+
+-h *host* | --host *host*
+Specifies the host name of the machine on which the HAWQ master database server is running. If not specified, reads from the environment variable `PGHOST` or defaults to localhost.
+
+-p *port* | --port *port*
+Specifies the TCP port on which the HAWQ master database server is listening for connections. If not specified, reads from the environment variable `PGPORT` or defaults to 5432.
+
+-U *username* | --username *username*
+The database role name to connect as. If not specified, reads from the environment variable `PGUSER` or defaults to the current system user name.
+
+-w | --no-password
+Never issue a password prompt. If the server requires password authentication and a password is not available by other means such as a `.pgpass` file, the connection attempt will fail. This option can be useful in batch jobs and scripts where no user is present to enter a password.
+
+-W | --password
+Force a password prompt.
+
+## Notes<a id="topic1__section6"></a>
+
+`vacuumdb` might need to connect several times to the master server, asking for a password each time. It is convenient to have a `~/.pgpass` file in such cases.
+
+## Examples<a id="topic1__section7"></a>
+
+To clean the database `test`:
+
+``` pre
+vacuumdb test
+```
+
+To clean and analyze a database named `bigdb`:
+
+``` pre
+vacuumdb --analyze bigdb
+```
+
+To clean a single table `foo` in a database named `mydb`, and analyze a single column `bar` of the table. Note the quotes around the table and column names to escape the parentheses from the shell:
+
+``` pre
+vacuumdb --analyze --verbose --table 'foo(bar)' mydb
+```
diff --git a/reference/cli/management_tools.html.md.erb b/reference/cli/management_tools.html.md.erb
new file mode 100644
index 0000000..c2127e5
--- /dev/null
+++ b/reference/cli/management_tools.html.md.erb
@@ -0,0 +1,63 @@
+---
+title: HAWQ Management Tools Reference
+---
+
+Reference information for command-line utilities available in HAWQ.
+
+- **[analyzedb](../../reference/cli/admin_utilities/analyzedb.html)**
+
+- **[createdb](../../reference/cli/client_utilities/createdb.html)**
+
+- **[createuser](../../reference/cli/client_utilities/createuser.html)**
+
+- **[dropdb](../../reference/cli/client_utilities/dropdb.html)**
+
+- **[dropuser](../../reference/cli/client_utilities/dropuser.html)**
+
+- **[gpfdist](../../reference/cli/admin_utilities/gpfdist.html)**
+
+- **[gplogfilter](../../reference/cli/admin_utilities/gplogfilter.html)**
+
+- **[gppkg](../../reference/cli/admin_utilities/gppkg.html)**
+
+- **[hawq activate](../../reference/cli/admin_utilities/hawqactivate.html)**
+
+- **[hawq check](../../reference/cli/admin_utilities/hawqcheck.html)**
+
+- **[hawq checkperf](../../reference/cli/admin_utilities/hawqcheckperf.html)**
+
+- **[hawq config](../../reference/cli/admin_utilities/hawqconfig.html)**
+
+- **[hawq extract](../../reference/cli/admin_utilities/hawqextract.html)**
+
+- **[hawq filespace](../../reference/cli/admin_utilities/hawqfilespace.html)**
+
+- **[hawq init](../../reference/cli/admin_utilities/hawqinit.html)**
+
+- **[hawq load](../../reference/cli/admin_utilities/hawqload.html)**
+
+- **[hawq restart](../../reference/cli/admin_utilities/hawqrestart.html)**
+
+- **[hawq scp](../../reference/cli/admin_utilities/hawqscp.html)**
+
+- **[hawq ssh](../../reference/cli/admin_utilities/hawqssh.html)**
+
+- **[hawq ssh-exkeys](../../reference/cli/admin_utilities/hawqssh-exkeys.html)**
+
+- **[hawq start](../../reference/cli/admin_utilities/hawqstart.html)**
+
+- **[hawq state](../../reference/cli/admin_utilities/hawqstate.html)**
+
+- **[hawq stop](../../reference/cli/admin_utilities/hawqstop.html)**
+
+- **[pg\_dump](../../reference/cli/client_utilities/pg_dump.html)**
+
+- **[pg\_dumpall](../../reference/cli/client_utilities/pg_dumpall.html)**
+
+- **[pg\_restore](../../reference/cli/client_utilities/pg_restore.html)**
+
+- **[psql](../../reference/cli/client_utilities/psql.html)**
+
+- **[vacuumdb](../../reference/cli/client_utilities/vacuumdb.html)**
+
+
diff --git a/reference/guc/guc_category-list.html.md.erb b/reference/guc/guc_category-list.html.md.erb
new file mode 100644
index 0000000..d5864fe
--- /dev/null
+++ b/reference/guc/guc_category-list.html.md.erb
@@ -0,0 +1,400 @@
+---
+title: Configuration Parameter Categories
+---
+
+Configuration parameters affect categories of server behaviors, such as resource consumption, query tuning, and authentication. The following sections describe HAWQ configuration parameter categories.
+
+
+## Append-Only Table Parameters<a id="topic_hfd_1tl_zp"></a>
+
+The following parameters configure the <span class="ph">append-only</span> tables feature of HAWQ.
+
+- [max\_appendonly\_tables](parameter_definitions.html#max_appendonly_tables)
+- [optimizer\_parts\_to\_force\_sort\_on\_insert](parameter_definitions.html#optimizer_parts_to_force_sort_on_insert)
+
+## Client Connection Default Parameters<a id="topic39"></a>
+
+These configuration parameters set defaults that are used for client connections.
+
+### Statement Behavior Parameters<a id="topic40"></a>
+
+- [check\_function\_bodies](parameter_definitions.html#check_function_bodies)
+- [default\_tablespace](parameter_definitions.html#default_tablespace)
+- [default\_transaction\_isolation](parameter_definitions.html#default_transaction_isolation)
+- [default\_transaction\_read\_only](parameter_definitions.html#default_transaction_read_only)
+- [search\_path](parameter_definitions.html#search_path)
+- [statement\_timeout](parameter_definitions.html#statement_timeout)
+- [vacuum\_freeze\_min\_age](parameter_definitions.html#vacuum_freeze_min_age)
+
+### Locale and Formatting Parameters<a id="topic41"></a>
+
+- [client\_encoding](parameter_definitions.html#client_encoding)
+- [DateStyle](parameter_definitions.html#DateStyle)
+- [extra\_float\_digits](parameter_definitions.html#extra_float_digits)
+- [IntervalStyle](parameter_definitions.html#IntervalStyle)
+- [lc\_collate](parameter_definitions.html#lc_collate)
+- [lc\_ctype](parameter_definitions.html#lc_ctype)
+- [lc\_messages](parameter_definitions.html#lc_messages)
+- [lc\_monetary](parameter_definitions.html#lc_monetary)
+- [lc\_numeric](parameter_definitions.html#lc_numeric)
+- [lc\_time](parameter_definitions.html#lc_time)
+- [TimeZone](parameter_definitions.html#TimeZone)
+
+### Other Client Default Parameters<a id="topic42"></a>
+
+- [dynamic\_library\_path](parameter_definitions.html#dynamic_library_path)
+- [explain\_pretty\_print](parameter_definitions.html#explain_pretty_print)
+- [local\_preload\_libraries](parameter_definitions.html#local_preload_libraries)
+
+## Connection and Authentication Parameters<a id="topic12"></a>
+
+These parameters control how clients connect and authenticate to HAWQ.
+
+### Connection Parameters<a id="topic13"></a>
+
+- [listen\_addresses](parameter_definitions.html#listen_addresses)
+- [max\_connections](parameter_definitions.html#max_connections)
+- [max\_prepared\_transactions](parameter_definitions.html#max_prepared_transactions)
+- [superuser\_reserved\_connections](parameter_definitions.html#superuser_reserved_connections)
+- [tcp\_keepalives\_count](parameter_definitions.html#tcp_keepalives_count)
+- [tcp\_keepalives\_idle](parameter_definitions.html#tcp_keepalives_idle)
+- [tcp\_keepalives\_interval](parameter_definitions.html#tcp_keepalives_interval)
+- [unix\_socket\_directory](parameter_definitions.html#unix_socket_directory)
+- [unix\_socket\_group](parameter_definitions.html#unix_socket_group)
+- [unix\_socket\_permissions](parameter_definitions.html#unix_socket_permissions)
+
+### Security and Authentication Parameters<a id="topic14"></a>
+
+- [authentication\_timeout](parameter_definitions.html#authentication_timeout)
+- [db\_user\_namespace](parameter_definitions.html#db_user_namespace)
+- [krb\_caseins\_users](parameter_definitions.html#krb_caseins_users)
+- [krb\_server\_keyfile](parameter_definitions.html#krb_server_keyfile)
+- [krb\_srvname](parameter_definitions.html#krb_srvname)
+- [password\_encryption](parameter_definitions.html#password_encryption)
+- [ssl](parameter_definitions.html#ssl)
+- [ssl\_ciphers](parameter_definitions.html#ssl_ciphers)
+
+## Database and Tablespace/Filespace Parameters<a id="topic47"></a>
+
+The following parameters configure the maximum number of databases, tablespaces, and filespaces allowed in a system.
+
+- [gp\_max\_tablespaces](parameter_definitions.html#gp_max_tablespaces)
+- [gp\_max\_filespaces](parameter_definitions.html#gp_max_filespaces)
+- [gp\_max\_databases](parameter_definitions.html#gp_max_databases)
+
+## Error Reporting and Logging Parameters<a id="topic29"></a>
+
+These configuration parameters control HAWQ logging.
+
+### Log Rotation<a id="topic30"></a>
+
+- [log\_rotation\_age](parameter_definitions.html#log_rotation_age)
+- [log\_rotation\_size](parameter_definitions.html#log_rotation_size)
+- [log\_truncate\_on\_rotation](parameter_definitions.html#log_truncate_on_rotation)
+
+### When to Log<a id="topic31"></a>
+
+- [client\_min\_messages](parameter_definitions.html#client_min_messages)
+- [log\_error\_verbosity](parameter_definitions.html#log_error_verbosity)
+- [log\_min\_duration\_statement](parameter_definitions.html#log_min_duration_statement)
+- [log\_min\_error\_statement](parameter_definitions.html#log_min_error_statement)
+- [log\_min\_messages](parameter_definitions.html#log_min_messages)
+- [optimizer\_minidump](parameter_definitions.html#optimizer_minidump)
+
+### What to Log<a id="topic32"></a>
+
+- [debug\_pretty\_print](parameter_definitions.html#debug_pretty_print)
+- [debug\_print\_parse](parameter_definitions.html#debug_print_parse)
+- [debug\_print\_parse](parameter_definitions.html#debug_print_parse)
+- [debug\_print\_plan](parameter_definitions.html#debug_print_plan)
+- [debug\_print\_prelim\_plan](parameter_definitions.html#debug_print_prelim_plan)
+- [debug\_print\_rewritten](parameter_definitions.html#debug_print_rewritten)
+- [debug\_print\_slice\_table](parameter_definitions.html#debug_print_slice_table)
+- [log\_autostats](parameter_definitions.html#log_autostats)
+- [log\_connections](parameter_definitions.html#log_connections)
+- [log\_disconnections](parameter_definitions.html#log_disconnections)
+- [log\_dispatch\_stats](parameter_definitions.html#log_dispatch_stats)
+- [log\_duration](parameter_definitions.html#log_duration)
+- [log\_executor\_stats](parameter_definitions.html#log_executor_stats)
+- [log\_hostname](parameter_definitions.html#log_hostname)
+- [log\_parser\_stats](parameter_definitions.html#log_parser_stats)
+- [log\_planner\_stats](parameter_definitions.html#log_planner_stats)
+- [log\_statement](parameter_definitions.html#log_statement)
+- [log\_statement\_stats](parameter_definitions.html#log_statement_stats)
+- [log\_timezone](parameter_definitions.html#log_timezone)
+- [gp\_debug\_linger](parameter_definitions.html#gp_debug_linger)
+- [gp\_log\_format](parameter_definitions.html#gp_log_format)
+- [gp\_max\_csv\_line\_length](parameter_definitions.html#gp_max_csv_line_length)
+- [gp\_reraise\_signal](parameter_definitions.html#gp_reraise_signal)
+
+## External Table Parameters<a id="topic45"></a>
+
+The following parameters configure the external tables feature of HAWQ.
+
+- [gp\_external\_enable\_exec](parameter_definitions.html#gp_external_enable_exec)
+- [gp\_external\_grant\_privileges](parameter_definitions.html#gp_external_grant_privileges)
+- [gp\_external\_max\_segs](parameter_definitions.html#gp_external_max_segs)
+- [gp\_reject\_percent\_threshold](parameter_definitions.html#gp_reject_percent_threshold)
+
+## GPORCA Parameters<a id="topic57"></a>
+
+These parameters control the usage of GPORCA by HAWQ. For information about GPORCA, see [About GPORCA](../../query/gporca/query-gporca-optimizer.html#topic_i4y_prl_vp).
+
+- [optimizer](parameter_definitions.html#optimizer)
+- [optimizer\_analyze\_root\_partition](parameter_definitions.html#optimizer_analyze_root_partition)
+- [optimizer\_minidump](parameter_definitions.html#optimizer_minidump)
+- [optimizer\_parts\_to\_force\_sort\_on\_insert](parameter_definitions.html#optimizer_parts_to_force_sort_on_insert)
+- [optimizer\_prefer\_scalar\_dqa\_multistage\_agg](parameter_definitions.html#optimizer_prefer_scalar_dqa_multistage_agg)
+
+## HAWQ Array Configuration Parameters<a id="topic49"></a>
+
+The parameters in this topic control the configuration of the HAWQ array and its components: segments, master, distributed transaction manager, master mirror, and interconnect.
+
+### Interconnect Configuration Parameters<a id="topic50"></a>
+
+- [gp\_interconnect\_fc\_method](parameter_definitions.html#gp_interconnect_fc_method)
+- [gp\_interconnect\_hash\_multiplier](parameter_definitions.html#gp_interconnect_hash_multiplier)
+- [gp\_interconnect\_queue\_depth](parameter_definitions.html#gp_interconnect_queue_depth)
+- [gp\_interconnect\_snd\_queue\_depth](parameter_definitions.html#gp_interconnect_snd_queue_depth)
+- [gp\_interconnect\_setup\_timeout](parameter_definitions.html#gp_interconnect_setup_timeout)
+- [gp\_interconnect\_type](parameter_definitions.html#gp_interconnect_type)
+- [gp\_max\_packet\_size](parameter_definitions.html#gp_max_packet_size)
+
+### Dispatch Configuration Parameters<a id="topic51"></a>
+
+- [gp\_cached\_segworkers\_threshold](parameter_definitions.html#gp_cached_segworkers_threshold)
+- [gp\_connections\_per\_thread](parameter_definitions.html#gp_connections_per_thread)
+- [gp\_enable\_direct\_dispatch](parameter_definitions.html#gp_enable_direct_dispatch)
+- [gp\_segment\_connect\_timeout](parameter_definitions.html#gp_segment_connect_timeout)
+- [gp\_set\_proc\_affinity](parameter_definitions.html#gp_set_proc_affinity)
+
+### Fault Operation Parameters<a id="topic52"></a>
+
+- [gp\_set\_read\_only](parameter_definitions.html#gp_set_read_only)
+
+### Filepace Parameters<a id="topic_ctl_sww_vv"></a>
+
+- [hawq\_dfs\_url](parameter_definitions.html#hawq_dfs_url)
+
+### Master Configuration Parameters<a id="topic_r4m_5ww_vv"></a>
+
+- [hawq\_master\_address\_host](parameter_definitions.html#hawq_master_address_host)
+- [hawq\_master\_address\_port](parameter_definitions.html#hawq_master_address_port)
+- [hawq\_master\_directory](parameter_definitions.html#hawq_master_directory)
+- [hawq\_master\_temp\_directory](parameter_definitions.html#hawq_master_temp_directory)
+
+### Read-Only Parameters<a id="topic54"></a>
+
+- [gp\_command\_count](parameter_definitions.html#gp_command_count)
+- [gp\_role](parameter_definitions.html#gp_role)
+- [gp\_session\_id](parameter_definitions.html#gp_session_id)
+
+### Segment Configuration Parameters<a id="topic_zgm_vww_vv"></a>
+
+- [hawq\_segment\_address\_port](parameter_definitions.html#hawq_segment_address_port)
+- [hawq\_segment\_directory](parameter_definitions.html#hawq_segment_directory)
+- [hawq\_segment\_temp\_directory](parameter_definitions.html#hawq_segment_temp_directory)
+
+## HAWQ PL/Java Extension Parameters<a id="topic56"></a>
+
+The parameters in this topic control the configuration of HAWQ PL/Java extensions.
+
+- [pljava\_classpath](parameter_definitions.html#pljava_classpath)
+- [pljava\_statement\_cache\_size](parameter_definitions.html#pljava_statement_cache_size)
+- [pljava\_release\_lingering\_savepoints](parameter_definitions.html#pljava_release_lingering_savepoints)
+- [pljava\_vmoptions](parameter_definitions.html#pljava_vmoptions)
+
+## HAWQ Resource Management Parameters<a id="hawq_resource_management"></a>
+
+The following configuration parameters configure the HAWQ resource management feature.
+
+- [hawq\_global\_rm\_type](parameter_definitions.html#hawq_global_rm_type)
+- [hawq\_re\_memory\_overcommit\_max](parameter_definitions.html#hawq_re_memory_overcommit_max)
+- [hawq\_rm\_cluster\_report\_period](parameter_definitions.html#hawq_rm_cluster_report)
+- [hawq\_rm\_force\_alterqueue\_cancel\_queued\_request](parameter_definitions.html#hawq_rm_force_alterqueue_cancel_queued_request)
+- [hawq\_rm\_master\_port](parameter_definitions.html#hawq_rm_master_port)
+- [hawq\_rm\_memory\_limit\_perseg](parameter_definitions.html#hawq_rm_memory_limit_perseg)
+- [hawq\_rm\_min\_resource\_perseg](parameter_definitions.html#hawq_rm_min_resource_perseg)
+- [hawq\_rm\_nresqueue\_limit](parameter_definitions.html#hawq_rm_nresqueue_limit)
+- [hawq\_rm\_nslice\_perseg\_limit](parameter_definitions.html#hawq_rm_nslice_perseg_limit)
+- [hawq\_rm\_nvcore\_limit\_perseg](parameter_definitions.html#hawq_rm_nvcore_limit_perseg)
+- [hawq\_rm\_nvseg\_perquery\_limit](parameter_definitions.html#hawq_rm_nvseg_perquery_limit)
+- [hawq\_rm\_nvseg\_perquery\_perseg\_limit](parameter_definitions.html#hawq_rm_nvseg_perquery_perseg_limit)
+- [hawq\_rm\_nvseg\_variance\_amon\_seg\_limit](parameter_definitions.html#hawq_rm_nvseg_variance_amon_seg_limit)
+- [hawq\_rm\_rejectrequest\_nseg\_limit](parameter_definitions.html#hawq_rm_rejectrequest_nseg_limit)
+- [hawq\_rm\_resource\_idle\_timeout](parameter_definitions.html#hawq_rm_resource_idle_timeout)
+- [hawq\_rm\_return\_percent\_on\_overcommit](parameter_definitions.html#hawq_rm_return_percent_on_overcommit)
+- [hawq\_rm\_segment\_heartbeat\_interval](parameter_definitions.html#hawq_rm_segment_heartbeat_interval)
+- [hawq\_rm\_segment\_port](parameter_definitions.html#hawq_rm_segment_port)
+- [hawq\_rm\_stmt\_nvseg](parameter_definitions.html#hawq_rm_stmt_nvseg)
+- [hawq\_rm\_stmt\_vseg\_memory](parameter_definitions.html#hawq_rm_stmt_vseg_memory)
+- [hawq\_rm\_tolerate\_nseg\_limit](parameter_definitions.html#hawq_rm_tolerate_nseg_limit)
+- [hawq\_rm\_yarn\_address](parameter_definitions.html#hawq_rm_yarn_address)
+- [hawq\_rm\_yarn\_app\_name](parameter_definitions.html#hawq_rm_yarn_app_name)
+- [hawq\_rm\_yarn\_queue\_name](parameter_definitions.html#hawq_rm_yarn_queue_name)
+- [hawq\_rm\_yarn\_scheduler\_address](parameter_definitions.html#hawq_rm_yarn_scheduler_address)
+
+## Lock Management Parameters<a id="topic43"></a>
+
+These configuration parameters set limits for locks and deadlocks.
+
+- [deadlock\_timeout](parameter_definitions.html#deadlock_timeout)
+- [max\_locks\_per\_transaction](parameter_definitions.html#max_locks_per_transaction)
+
+## Past PostgreSQL Version Compatibility Parameters<a id="topic48"></a>
+
+The following parameters provide compatibility with older PostgreSQL versions. You do not need to change these parameters in HAWQ.
+
+- [add\_missing\_from](parameter_definitions.html#add_missing_from)
+- [array\_nulls](parameter_definitions.html#array_nulls)
+- [backslash\_quote](parameter_definitions.html#backslash_quote)
+- [escape\_string\_warning](parameter_definitions.html#escape_string_warning)
+- [regex\_flavor](parameter_definitions.html#regex_flavor)
+- [standard\_conforming\_strings](parameter_definitions.html#standard_conforming_strings)
+- [transform\_null\_equals](parameter_definitions.html#transform_null_equals)
+
+## Query Tuning Parameters<a id="topic21"></a>
+
+These parameters control aspects of SQL query processing such as query operators and operator settings and statistics sampling.
+
+### Legacy Query Optimizer Operator Control Parameters<a id="topic22"></a>
+
+The following parameters control the types of plan operations the legacy query optimizer can use. Enable or disable plan operations to force the legacy optimizer to choose a different plan. This is useful for testing and comparing query performance using different plan types.
+
+- [enable\_bitmapscan](parameter_definitions.html#enable_bitmapscan)
+- [enable\_groupagg](parameter_definitions.html#enable_groupagg)
+- [enable\_hashagg](parameter_definitions.html#enable_hashagg)
+- [enable\_hashjoin](parameter_definitions.html#enable_hashjoin)
+- [enable\_indexscan](parameter_definitions.html#enable_indexscan)
+- [enable\_mergejoin](parameter_definitions.html#enable_mergejoin)
+- [enable\_nestloop](parameter_definitions.html#enable_nestloop)
+- [enable\_seqscan](parameter_definitions.html#enable_seqscan)
+- [enable\_sort](parameter_definitions.html#enable_sort)
+- [enable\_tidscan](parameter_definitions.html#enable_tidscan)
+- [gp\_enable\_agg\_distinct](parameter_definitions.html#gp_enable_agg_distinct)
+- [gp\_enable\_agg\_distinct\_pruning](parameter_definitions.html#gp_enable_agg_distinct_pruning)
+- [gp\_enable\_direct\_dispatch](parameter_definitions.html#gp_enable_direct_dispatch)
+- [gp\_enable\_fallback\_plan](parameter_definitions.html#gp_enable_fallback_plan)
+- [gp\_enable\_fast\_sri](parameter_definitions.html#gp_enable_fast_sri)
+- [gp\_enable\_groupext\_distinct\_gather](parameter_definitions.html#gp_enable_groupext_distinct_gather)
+- [gp\_enable\_groupext\_distinct\_pruning](parameter_definitions.html#gp_enable_groupext_distinct_pruning)
+- [gp\_enable\_multiphase\_agg](parameter_definitions.html#gp_enable_multiphase_agg)
+- [gp\_enable\_predicate\_propagation](parameter_definitions.html#gp_enable_predicate_propagation)
+- [gp\_enable\_preunique](parameter_definitions.html#gp_enable_preunique)
+- [gp\_enable\_sequential\_window\_plans](parameter_definitions.html#gp_enable_sequential_window_plans)
+- [gp\_enable\_sort\_distinct](parameter_definitions.html#gp_enable_sort_distinct)
+- [gp\_enable\_sort\_limit](parameter_definitions.html#gp_enable_sort_limit)
+
+### Legacy Query Optimizer Costing Parameters<a id="topic23"></a>
+
+**Warning:** Do not adjust these query costing parameters. They are tuned to reflect HAWQ hardware configurations and typical workloads. All of these parameters are related. Changing one without changing the others can have adverse affects on performance.
+
+- [cpu\_index\_tuple\_cost](parameter_definitions.html#cpu_index_tuple_cost)
+- [cpu\_operator\_cost](parameter_definitions.html#cpu_operator_cost)
+- [cpu\_tuple\_cost](parameter_definitions.html#cpu_tuple_cost)
+- [cursor\_tuple\_fraction](parameter_definitions.html#cursor_tuple_fraction)
+- [effective\_cache\_size](parameter_definitions.html#effective_cache_size)
+- [gp\_motion\_cost\_per\_row](parameter_definitions.html#gp_motion_cost_per_row)
+- [gp\_segments\_for\_planner](parameter_definitions.html#gp_segments_for_planner)
+- [random\_page\_cost](parameter_definitions.html#random_page_cost)
+- [seq\_page\_cost](parameter_definitions.html#seq_page_cost)
+
+### Database Statistics Sampling Parameters<a id="topic24"></a>
+
+These parameters adjust the amount of data sampled by an `ANALYZE` operation. Adjusting these parameters affects statistics collection system-wide. You can configure statistics collection on particular tables and columns by using the `ALTER TABLE` `SET STATISTICS` clause. See [About Database Statistics](../../datamgmt/about_statistics.html).
+
+- [default\_statistics\_target](parameter_definitions.html#default_statistics_target)
+- [gp\_analyze\_relative\_error](parameter_definitions.html#gp_analyze_relative_error)
+
+### Sort Operator Configuration Parameters<a id="topic25"></a>
+
+- [gp\_enable\_sort\_distinct](parameter_definitions.html#gp_enable_sort_distinct)
+- [gp\_enable\_sort\_limit](parameter_definitions.html#gp_enable_sort_limit)
+
+### Aggregate Operator Configuration Parameters<a id="topic26"></a>
+
+- [gp\_enable\_agg\_distinct](parameter_definitions.html#gp_enable_agg_distinct)
+- [gp\_enable\_agg\_distinct\_pruning](parameter_definitions.html#gp_enable_agg_distinct_pruning)
+- [gp\_enable\_multiphase\_agg](parameter_definitions.html#gp_enable_multiphase_agg)
+- [gp\_enable\_preunique](parameter_definitions.html#gp_enable_preunique)
+- [gp\_enable\_groupext\_distinct\_gather](parameter_definitions.html#gp_enable_groupext_distinct_gather)
+- [gp\_enable\_groupext\_distinct\_pruning](parameter_definitions.html#gp_enable_groupext_distinct_pruning)
+- [gp\_workfile\_compress\_algorithm](parameter_definitions.html#gp_workfile_compress_algorithm)
+
+### Join Operator Configuration Parameters<a id="topic27"></a>
+
+- [join\_collapse\_limit](parameter_definitions.html#join_collapse_limit)
+- [gp\_adjust\_selectivity\_for\_outerjoins](parameter_definitions.html#gp_adjust_selectivity_for_outerjoins)
+- [gp\_hashjoin\_tuples\_per\_bucket](parameter_definitions.html#gp_hashjoin_tuples_per_bucket)
+- [gp\_statistics\_use\_fkeys](parameter_definitions.html#gp_statistics_use_fkeys)
+- [gp\_workfile\_compress\_algorithm](parameter_definitions.html#gp_workfile_compress_algorithm)
+
+### Other Legacy Query Optimizer Configuration Parameters<a id="topic28"></a>
+
+- [from\_collapse\_limit](parameter_definitions.html#from_collapse_limit)
+- [gp\_enable\_predicate\_propagation](parameter_definitions.html#gp_enable_predicate_propagation)
+- [gp\_max\_plan\_size](parameter_definitions.html#gp_max_plan_size)
+- [gp\_statistics\_pullup\_from\_child\_partition](parameter_definitions.html#gp_statistics_pullup_from_child_partition)
+
+## Statistics Collection Parameters<a id="statistics_collection"></a>
+
+### Automatic Statistics Collection<a id="topic_qvz_nz3_yv"></a>
+
+When automatic statistics collection is enabled, you can run `ANALYZE` automatically in the same transaction as an `INSERT`, `UPDATE`, `DELETE`, `COPY` or `CREATE TABLE...AS SELECT` statement when a certain threshold of rows is affected (`on_change`), or when a newly generated table has no statistics (`on_no_stats`). To enable this feature, set the following server configuration parameters in your HAWQ `hawq-site.xml` file by using the `hawq config` utility and restart HAWQ:
+
+- [gp\_autostats\_mode](parameter_definitions.html#gp_autostats_mode)
+- [log\_autostats](parameter_definitions.html#log_autostats)
+
+### Runtime Statistics Collection Parameters<a id="topic37"></a>
+
+These parameters control the server statistics collection feature. When statistics collection is enabled, you can access the statistics data using the *pg\_stat* and *pg\_statio* family of system catalog views.
+
+- [track\_activities](parameter_definitions.html#track_activities)
+- [track\_counts](parameter_definitions.html#track_counts)
+- [update\_process\_title](parameter_definitions.html#update_process_title)
+
+## System Resource Consumption Parameters<a id="topic15"></a>
+
+These parameters set the limits for system resources consumed by HAWQ.
+
+### Memory Consumption Parameters<a id="topic16"></a>
+
+These parameters control system memory usage. You can adjust `hawq_re_memory_overcommit_max` to avoid running out of memory at the segment hosts during query processing. See also [HAWQ Resource Management](#hawq_resource_management).
+
+- [hawq\_re\_memory\_overcommit\_max](parameter_definitions.html#hawq_re_memory_overcommit_max)
+- [gp\_vmem\_protect\_segworker\_cache\_limit](parameter_definitions.html#gp_vmem_protect_segworker_cache_limit)
+- [gp\_workfile\_limit\_files\_per\_query](parameter_definitions.html#gp_workfile_limit_files_per_query)
+- [gp\_workfile\_limit\_per\_query](parameter_definitions.html#gp_workfile_limit_per_query)
+- [gp\_workfile\_limit\_per\_segment](parameter_definitions.html#gp_workfile_limit_per_segment)
+- [max\_stack\_depth](parameter_definitions.html#max_stack_depth)
+- [shared\_buffers](parameter_definitions.html#shared_buffers)
+- [temp\_buffers](parameter_definitions.html#temp_buffers)
+
+### Free Space Map Parameters<a id="topic17"></a>
+
+These parameters control the sizing of the *free space map, which contains* expired rows. Use `VACUUM` to reclaim the free space map disk space.
+
+- [max\_fsm\_pages](parameter_definitions.html#max_fsm_pages)
+- [max\_fsm\_relations](parameter_definitions.html#max_fsm_relations)
+
+### OS Resource Parameters<a id="topic18"></a>
+
+- [max\_files\_per\_process](parameter_definitions.html#max_files_per_process)
+- [shared\_preload\_libraries](parameter_definitions.html#shared_preload_libraries)
+
+### Cost-Based Vacuum Delay Parameters<a id="topic19"></a>
+
+**Warning:** Avoid using cost-based vacuum delay because it runs asynchronously among the segment instances. The vacuum cost limit and delay is invoked at the segment level without taking into account the state of the entire HAWQ array
+
+You can configure the execution cost of `VACUUM` and `ANALYZE` commands to reduce the I/O impact on concurrent database activity. When the accumulated cost of I/O operations reaches the limit, the process performing the operation sleeps for a while, Then resets the counter and continues execution
+
+- [vacuum\_cost\_delay](parameter_definitions.html#vacuum_cost_delay)
+- [vacuum\_cost\_limit](parameter_definitions.html#vacuum_cost_limit)
+- [vacuum\_cost\_page\_dirty](parameter_definitions.html#vacuum_cost_page_dirty)
+- [vacuum\_cost\_page\_miss](parameter_definitions.html#vacuum_cost_page_miss)
+
+### Transaction ID Management Parameters<a id="topic20"></a>
+
+- [xid\_stop\_limit](parameter_definitions.html#xid_stop_limit)
+
+
diff --git a/reference/guc/guc_config.html.md.erb b/reference/guc/guc_config.html.md.erb
new file mode 100644
index 0000000..61d1db8
--- /dev/null
+++ b/reference/guc/guc_config.html.md.erb
@@ -0,0 +1,77 @@
+---
+title: About Server Configuration Parameters
+---
+
+There are many HAWQ server configuration parameters that affect the behavior of the HAWQ system. Many of these configuration parameters have the same names, settings, and behaviors as in a regular PostgreSQL database system.
+
+- [Parameter Types and Values](#topic_vsn_22l_z4) describes the parameter data types and values.
+- [Setting Parameters](#topic_cyz_p2l_z4) describes limitations on who can change them and where or when they can be set.
+- [Configuration Parameter Categories](guc_category-list.html#guc-cat-list) organizes parameters by functionality.
+- [Configuration Parameters](parameter_definitions.html) lists the parameter descriptions in alphabetic order.
+
+## Parameter Types and Values<a id="topic_vsn_22l_z4"></a>
+
+All parameter names are case-insensitive. Every parameter takes a value of one of four types: `Boolean`, `integer`, `floating point`, or `string`. Boolean values may be written as `ON`, `OFF`, `TRUE`, `FALSE`, `YES`, `NO`, `1`, `0` (all case-insensitive).
+
+Some settings specify a memory size or time value. Each of these has an implicit unit, which is either kilobytes, blocks (typically eight kilobytes), milliseconds, seconds, or minutes. Valid memory size units are `kB` (kilobytes), `MB` (megabytes), and `GB` (gigabytes). Valid time units are `ms` (milliseconds), `s` (seconds), `min` (minutes), `h` (hours), and `d` (days). Note that the multiplier for memory units is 1024, not 1000. A valid time expression contains a number and a unit. When specifying a memory or time unit using the `SET` command, enclose the value in quotes. For example:
+
+``` pre
+SET default_statement_mem TO '200MB';
+```
+
+**Note:** There is no space between the value and the unit names.
+
+## Setting Parameters<a id="topic_cyz_p2l_z4"></a>
+
+Many of the configuration parameters have limitations on who can change them and where or when they can be set. For example, to change certain parameters, you must be a HAWQ superuser. Other parameters require a restart of the system for the changes to take effect. A parameter that is classified as *session* can be set at the system level (in the `hawq-site.xml` file), at the database-level (using `ALTER DATABASE`), at the role-level (using `ALTER ROLE`), or at the session-level (using `SET`). System parameters can only be set by using the `hawq config` utility or by directly modifying a `hawq-site.xml` file.
+
+By design, all HAWQ instances (including master and segments) host identical `hawq-site.xml` files. Using a common `hawq-site.xml` file across all HAWQ instances simplifies configuration of the cluster. Within each `hawq-site.xml` configuration file, some parameters are considered *segment* parameters, meaning that each segment instance looks to its own `hawq-site.xml` file to get the value of that parameter. By convention, these parameter names begin with the string `hawq_segment`. Others parameters are considered *master* parameters. By convention, these parameter names begin with the string `hawq_master`. Master parameters are only applied at the master instance and ignored by segments.
+
+**Note:** If you use the `hawq config` utility to set configuration parameter values in `hawq-site.xml`, the utility synchronizes all configuration files. Any manual modifications that you made to individual `hawq-site.xml` files may be lost.
+
+This table describes the values in the Set Classifications column of the table in the description of a server configuration parameter.
+
+<a id="topic_cyz_p2l_z4__ih389119"></a>
+
+<table>
+<caption><span class="tablecap">Table 1. Set Classifications</span></caption>
+<colgroup>
+<col width="50%" />
+<col width="50%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Set Classification</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>master or local</td>
+<td>A <em>master</em> parameter must be set in the <code class="ph codeph">hawq-site.xml</code> file of the HAWQ master instance. The value for this parameter is then either passed to (or ignored by) the segments at run time.
+<p>A <em>local</em> parameter is also defined in the <code class="ph codeph">hawq-site.xml</code> file of the master AND every segment instance. Each HAWQ instance looks to its own configuration to get the value for the parameter.</p>
+<p>We recommend that you use the same configuration parameter values across all HAWQ instances to maintain a single, consistent <code class="ph codeph">hawq-site.xml</code> configuration file. The <code class="ph codeph">hawq config</code> utility will enforce this consistency.</p>
+<p>Changes to master or local parameters always require a system restart for changes to take effect.</p></td>
+</tr>
+<tr class="even">
+<td>session or system</td>
+<td><em>Session</em> parameters can be changed on the fly within a database session, and can have a hierarchy of settings: at the system level (through <code class="ph codeph">hawq-site.xml</code>or <code class="ph codeph">hawq config</code> utility), at the database level (<code class="ph codeph">ALTER DATABASE...SET</code>), at the role level (<code class="ph codeph">ALTER ROLE...SET</code>), or at the session level (<code class="ph codeph">SET</code>). If the parameter is set at multiple levels, then the most granular setting takes precedence (for example, session overrides role, role overrides database, and database overrides system).
+<p>A <em>system</em> parameter can only be changed via <code class="ph codeph">hawq config</code> utility or the <code class="ph codeph">hawq-site.xml</code> file(s).</p></td>
+</tr>
+<tr class="odd">
+<td>restart or reload</td>
+<td>When changing parameter values in the <code class="ph codeph">hawq-site.xml</code> file(s), some require a <em>restart</em> of HAWQ for the change to take effect. Other parameter values can be refreshed by just reloading the configuration file (using <code class="ph codeph">hawq stop object -u</code>), and do not require stopping the system.</td>
+</tr>
+<tr class="even">
+<td>superuser</td>
+<td>These session parameters can only be set by a database superuser. Regular database users cannot set this parameter.</td>
+</tr>
+<tr class="odd">
+<td>read only</td>
+<td>These parameters are not settable by database users or superusers. The current value of the parameter can be shown but not altered.</td>
+</tr>
+</tbody>
+</table>
+
+
+
diff --git a/reference/guc/parameter_definitions.html.md.erb b/reference/guc/parameter_definitions.html.md.erb
new file mode 100644
index 0000000..0503dd8
--- /dev/null
+++ b/reference/guc/parameter_definitions.html.md.erb
@@ -0,0 +1,3053 @@
+---
+title: Configuration Parameters
+---
+
+Descriptions of the HAWQ server configuration parameters listed alphabetically.
+
+- **[add\_missing\_from](../../reference/guc/parameter_definitions.html#add_missing_from)**
+
+- **[application\_name](../../reference/guc/parameter_definitions.html#application_name)**
+
+- **[array\_nulls](../../reference/guc/parameter_definitions.html#array_nulls)**
+
+- **[authentication\_timeout](../../reference/guc/parameter_definitions.html#authentication_timeout)**
+
+- **[backslash\_quote](../../reference/guc/parameter_definitions.html#backslash_quote)**
+
+- **[block\_size](../../reference/guc/parameter_definitions.html#block_size)**
+
+- **[bonjour\_name](../../reference/guc/parameter_definitions.html#bonjour_name)**
+
+- **[check\_function\_bodies](../../reference/guc/parameter_definitions.html#check_function_bodies)**
+
+- **[client\_encoding](../../reference/guc/parameter_definitions.html#client_encoding)**
+
+- **[client\_min\_messages](../../reference/guc/parameter_definitions.html#client_min_messages)**
+
+- **[cpu\_index\_tuple\_cost](../../reference/guc/parameter_definitions.html#cpu_index_tuple_cost)**
+
+- **[cpu\_operator\_cost](../../reference/guc/parameter_definitions.html#cpu_operator_cost)**
+
+- **[cpu\_tuple\_cost](../../reference/guc/parameter_definitions.html#cpu_tuple_cost)**
+
+- **[cursor\_tuple\_fraction](../../reference/guc/parameter_definitions.html#cursor_tuple_fraction)**
+
+- **[custom\_variable\_classes](../../reference/guc/parameter_definitions.html#custom_variable_classes)**
+
+- **[DateStyle](../../reference/guc/parameter_definitions.html#DateStyle)**
+
+- **[db\_user\_namespace](../../reference/guc/parameter_definitions.html#db_user_namespace)**
+
+- **[deadlock\_timeout](../../reference/guc/parameter_definitions.html#deadlock_timeout)**
+
+- **[debug\_assertions](../../reference/guc/parameter_definitions.html#debug_assertions)**
+
+- **[debug\_pretty\_print](../../reference/guc/parameter_definitions.html#debug_pretty_print)**
+
+- **[debug\_print\_parse](../../reference/guc/parameter_definitions.html#debug_print_parse)**
+
+- **[debug\_print\_plan](../../reference/guc/parameter_definitions.html#debug_print_plan)**
+
+- **[debug\_print\_prelim\_plan](../../reference/guc/parameter_definitions.html#debug_print_prelim_plan)**
+
+- **[debug\_print\_rewritten](../../reference/guc/parameter_definitions.html#debug_print_rewritten)**
+
+- **[debug\_print\_slice\_table](../../reference/guc/parameter_definitions.html#debug_print_slice_table)**
+
+- **[default\_hash\_table\_bucket\_number](../../reference/guc/parameter_definitions.html#topic_fqj_4fd_kv)**
+
+- **[default\_statistics\_target](../../reference/guc/parameter_definitions.html#default_statistics_target)**
+
+- **[default\_tablespace](../../reference/guc/parameter_definitions.html#default_tablespace)**
+
+- **[default\_transaction\_isolation](../../reference/guc/parameter_definitions.html#default_transaction_isolation)**
+
+- **[default\_transaction\_read\_only](../../reference/guc/parameter_definitions.html#default_transaction_read_only)**
+
+- **[dfs\_url](../../reference/guc/parameter_definitions.html#dfs_url)**
+
+- **[dynamic\_library\_path](../../reference/guc/parameter_definitions.html#dynamic_library_path)**
+
+- **[effective\_cache\_size](../../reference/guc/parameter_definitions.html#effective_cache_size)**
+
+- **[enable\_bitmapscan](../../reference/guc/parameter_definitions.html#enable_bitmapscan)**
+
+- **[enable\_groupagg](../../reference/guc/parameter_definitions.html#enable_groupagg)**
+
+- **[enable\_hashagg](../../reference/guc/parameter_definitions.html#enable_hashagg)**
+
+- **[enable\_hashjoin](../../reference/guc/parameter_definitions.html#enable_hashjoin)**
+
+- **[enable\_indexscan](../../reference/guc/parameter_definitions.html#enable_indexscan)**
+
+- **[enable\_mergejoin](../../reference/guc/parameter_definitions.html#enable_mergejoin)**
+
+- **[enable\_nestloop](../../reference/guc/parameter_definitions.html#enable_nestloop)**
+
+- **[enable\_seqscan](../../reference/guc/parameter_definitions.html#enable_seqscan)**
+
+- **[enable\_sort](../../reference/guc/parameter_definitions.html#enable_sort)**
+
+- **[enable\_tidscan](../../reference/guc/parameter_definitions.html#enable_tidscan)**
+
+- **[escape\_string\_warning](../../reference/guc/parameter_definitions.html#escape_string_warning)**
+
+- **[explain\_pretty\_print](../../reference/guc/parameter_definitions.html#explain_pretty_print)**
+
+- **[extra\_float\_digits](../../reference/guc/parameter_definitions.html#extra_float_digits)**
+
+- **[from\_collapse\_limit](../../reference/guc/parameter_definitions.html#from_collapse_limit)**
+
+- **[gp\_adjust\_selectivity\_for\_outerjoins](../../reference/guc/parameter_definitions.html#gp_adjust_selectivity_for_outerjoins)**
+
+- **[gp\_analyze\_relative\_error](../../reference/guc/parameter_definitions.html#gp_analyze_relative_error)**
+
+- **[gp\_autostats\_mode](../../reference/guc/parameter_definitions.html#gp_autostats_mode)**
+
+- **[gp\_autostats\_on\_change\_threshhold](../../reference/guc/parameter_definitions.html#topic_imj_zhf_gw)**
+
+- **[gp\_backup\_directIO](../../reference/guc/parameter_definitions.html#gp_backup_directIO)**
+
+- **[gp\_backup\_directIO\_read\_chunk\_mb](../../reference/guc/parameter_definitions.html#gp_backup_directIO_read_chunk_mb)**
+
+- **[gp\_cached\_segworkers\_threshold](../../reference/guc/parameter_definitions.html#gp_cached_segworkers_threshold)**
+
+- **[gp\_command\_count](../../reference/guc/parameter_definitions.html#gp_command_count)**
+
+- **[gp\_connections\_per\_thread](../../reference/guc/parameter_definitions.html#gp_connections_per_thread)**
+
+- **[gp\_debug\_linger](../../reference/guc/parameter_definitions.html#gp_debug_linger)**
+
+- **[gp\_dynamic\_partition\_pruning](../../reference/guc/parameter_definitions.html#gp_dynamic_partition_pruning)**
+
+- **[gp\_enable\_agg\_distinct](../../reference/guc/parameter_definitions.html#gp_enable_agg_distinct)**
+
+- **[gp\_enable\_agg\_distinct\_pruning](../../reference/guc/parameter_definitions.html#gp_enable_agg_distinct_pruning)**
+
+- **[gp\_enable\_direct\_dispatch](../../reference/guc/parameter_definitions.html#gp_enable_direct_dispatch)**
+
+- **[gp\_enable\_fallback\_plan](../../reference/guc/parameter_definitions.html#gp_enable_fallback_plan)**
+
+- **[gp\_enable\_fast\_sri](../../reference/guc/parameter_definitions.html#gp_enable_fast_sri)**
+
+- **[gp\_enable\_groupext\_distinct\_gather](../../reference/guc/parameter_definitions.html#gp_enable_groupext_distinct_gather)**
+
+- **[gp\_enable\_groupext\_distinct\_pruning](../../reference/guc/parameter_definitions.html#gp_enable_groupext_distinct_pruning)**
+
+- **[gp\_enable\_multiphase\_agg](../../reference/guc/parameter_definitions.html#gp_enable_multiphase_agg)**
+
+- **[gp\_enable\_predicate\_propagation](../../reference/guc/parameter_definitions.html#gp_enable_predicate_propagation)**
+
+- **[gp\_enable\_preunique](../../reference/guc/parameter_definitions.html#gp_enable_preunique)**
+
+- **[gp\_enable\_sequential\_window\_plans](../../reference/guc/parameter_definitions.html#gp_enable_sequential_window_plans)**
+
+- **[gp\_enable\_sort\_distinct](../../reference/guc/parameter_definitions.html#gp_enable_sort_distinct)**
+
+- **[gp\_enable\_sort\_limit](../../reference/guc/parameter_definitions.html#gp_enable_sort_limit)**
+
+- **[gp\_external\_enable\_exec](../../reference/guc/parameter_definitions.html#gp_external_enable_exec)**
+
+- **[gp\_external\_grant\_privileges](../../reference/guc/parameter_definitions.html#gp_external_grant_privileges)**
+
+- **[gp\_external\_max\_segs](../../reference/guc/parameter_definitions.html#gp_external_max_segs)**
+
+- **[gp\_filerep\_tcp\_keepalives\_count](../../reference/guc/parameter_definitions.html#gp_filerep_tcp_keepalives_count)**
+
+- **[gp\_filerep\_tcp\_keepalives\_idle](../../reference/guc/parameter_definitions.html#gp_filerep_tcp_keepalives_idle)**
+
+- **[gp\_filerep\_tcp\_keepalives\_interval](../../reference/guc/parameter_definitions.html#gp_filerep_tcp_keepalives_interval)**
+
+- **[gp\_hashjoin\_tuples\_per\_bucket](../../reference/guc/parameter_definitions.html#gp_hashjoin_tuples_per_bucket)**
+
+- **[gp\_idf\_deduplicate](../../reference/guc/parameter_definitions.html#gp_idf_deduplicate)**
+
+- **[gp\_interconnect\_fc\_method](../../reference/guc/parameter_definitions.html#gp_interconnect_fc_method)**
+
+- **[gp\_interconnect\_hash\_multiplier](../../reference/guc/parameter_definitions.html#gp_interconnect_hash_multiplier)**
+
+- **[gp\_interconnect\_queue\_depth](../../reference/guc/parameter_definitions.html#gp_interconnect_queue_depth)**
+
+- **[gp\_interconnect\_setup\_timeout](../../reference/guc/parameter_definitions.html#gp_interconnect_setup_timeout)**
+
+- **[gp\_interconnect\_snd\_queue\_depth](../../reference/guc/parameter_definitions.html#gp_interconnect_snd_queue_depth)**
+
+- **[gp\_interconnect\_type](../../reference/guc/parameter_definitions.html#gp_interconnect_type)**
+
+- **[gp\_log\_format](../../reference/guc/parameter_definitions.html#gp_log_format)**
+
+- **[gp\_max\_csv\_line\_length](../../reference/guc/parameter_definitions.html#gp_max_csv_line_length)**
+
+- **[gp\_max\_databases](../../reference/guc/parameter_definitions.html#gp_max_databases)**
+
+- **[gp\_max\_filespaces](../../reference/guc/parameter_definitions.html#gp_max_filespaces)**
+
+- **[gp\_max\_packet\_size](../../reference/guc/parameter_definitions.html#gp_max_packet_size)**
+
+- **[gp\_max\_plan\_size](../../reference/guc/parameter_definitions.html#gp_max_plan_size)**
+
+- **[gp\_max\_tablespaces](../../reference/guc/parameter_definitions.html#gp_max_tablespaces)**
+
+- **[gp\_motion\_cost\_per\_row](../../reference/guc/parameter_definitions.html#gp_motion_cost_per_row)**
+
+- **[gp\_reject\_percent\_threshold](../../reference/guc/parameter_definitions.html#gp_reject_percent_threshold)**
+
+- **[gp\_reraise\_signal](../../reference/guc/parameter_definitions.html#gp_reraise_signal)**
+
+- **[gp\_role](../../reference/guc/parameter_definitions.html#gp_role)**
+
+- **[gp\_safefswritesize](../../reference/guc/parameter_definitions.html#gp_safefswritesize)**
+
+- **[gp\_segment\_connect\_timeout](../../reference/guc/parameter_definitions.html#gp_segment_connect_timeout)**
+
+- **[gp\_segments\_for\_planner](../../reference/guc/parameter_definitions.html#gp_segments_for_planner)**
+
+- **[gp\_session\_id](../../reference/guc/parameter_definitions.html#gp_session_id)**
+
+- **[gp\_set\_proc\_affinity](../../reference/guc/parameter_definitions.html#gp_set_proc_affinity)**
+
+- **[gp\_set\_read\_only](../../reference/guc/parameter_definitions.html#gp_set_read_only)**
+
+- **[gp\_statistics\_pullup\_from\_child\_partition](../../reference/guc/parameter_definitions.html#gp_statistics_pullup_from_child_partition)**
+
+- **[gp\_statistics\_use\_fkeys](../../reference/guc/parameter_definitions.html#gp_statistics_use_fkeys)**
+
+- **[gp\_vmem\_protect\_segworker\_cache\_limit](../../reference/guc/parameter_definitions.html#gp_vmem_protect_segworker_cache_limit)**
+
+- **[gp\_workfile\_checksumming](../../reference/guc/parameter_definitions.html#gp_workfile_checksumming)**
+
+- **[gp\_workfile\_compress\_algorithm](../../reference/guc/parameter_definitions.html#gp_workfile_compress_algorithm)**
+
+- **[gp\_workfile\_limit\_files\_per\_query](../../reference/guc/parameter_definitions.html#gp_workfile_limit_files_per_query)**
+
+- **[gp\_workfile\_limit\_per\_query](../../reference/guc/parameter_definitions.html#gp_workfile_limit_per_query)**
+
+- **[gp\_workfile\_limit\_per\_segment](../../reference/guc/parameter_definitions.html#gp_workfile_limit_per_segment)**
+
+- **[hawq\_dfs\_url](../../reference/guc/parameter_definitions.html#hawq_dfs_url)**
+
+- **[hawq\_global\_rm\_type](../../reference/guc/parameter_definitions.html#hawq_global_rm_type)**
+
+- **[hawq\_master\_address\_host](../../reference/guc/parameter_definitions.html#hawq_master_address_host)**
+
+- **[hawq\_master\_address\_port](../../reference/guc/parameter_definitions.html#hawq_master_address_port)**
+
+- **[hawq\_master\_directory](../../reference/guc/parameter_definitions.html#hawq_master_directory)**
+
+- **[hawq\_master\_temp\_directory](../../reference/guc/parameter_definitions.html#hawq_master_temp_directory)**
+
+- **[hawq\_re\_memory\_overcommit\_max](../../reference/guc/parameter_definitions.html#hawq_re_memory_overcommit_max)**
+
+- **[hawq\_rm\_cluster\_report\_period](../../reference/guc/parameter_definitions.html#hawq_rm_cluster_report)**
+
+- **[hawq\_rm\_force\_alterqueue\_cancel\_queued\_request](../../reference/guc/parameter_definitions.html#hawq_rm_force_alterqueue_cancel_queued_request)**
+
+- **[hawq\_rm\_master\_port](../../reference/guc/parameter_definitions.html#hawq_rm_master_port)**
+
+- **[hawq\_rm\_memory\_limit\_perseg](../../reference/guc/parameter_definitions.html#hawq_rm_memory_limit_perseg)**
+
+- **[hawq\_rm\_min\_resource\_perseg](../../reference/guc/parameter_definitions.html#hawq_rm_min_resource_perseg)**
+
+- **[hawq\_rm\_nresqueue\_limit](../../reference/guc/parameter_definitions.html#hawq_rm_nresqueue_limit)**
+
+- **[hawq\_rm\_nslice\_perseg\_limit](../../reference/guc/parameter_definitions.html#hawq_rm_nslice_perseg_limit)**
+
+- **[hawq\_rm\_nvcore\_limit\_perseg](../../reference/guc/parameter_definitions.html#hawq_rm_nvcore_limit_perseg)**
+
+- **[hawq\_rm\_nvseg\_perquery\_limit](../../reference/guc/parameter_definitions.html#hawq_rm_nvseg_perquery_limit)**
+
+- **[hawq\_rm\_nvseg\_perquery\_perseg\_limit](../../reference/guc/parameter_definitions.html#hawq_rm_nvseg_perquery_perseg_limit)**
+
+- **[hawq\_rm\_nvseg\_variance\_amon\_seg\_limit](../../reference/guc/parameter_definitions.html#hawq_rm_nvseg_variance_amon_seg_limit)**
+
+- **[hawq\_rm\_rejectrequest\_nseg\_limit](../../reference/guc/parameter_definitions.html#hawq_rm_rejectrequest_nseg_limit)**
+
+- **[hawq\_rm\_resource\_idle\_timeout](../../reference/guc/parameter_definitions.html#hawq_rm_resource_idle_timeout)**
+
+- **[hawq\_rm\_return\_percent\_on\_overcommit](../../reference/guc/parameter_definitions.html#hawq_rm_return_percent_on_overcommit)**
+
+- **[hawq\_rm\_segment\_heartbeat\_interval](../../reference/guc/parameter_definitions.html#hawq_rm_segment_heartbeat_interval)**
+
+- **[hawq\_rm\_segment\_port](../../reference/guc/parameter_definitions.html#hawq_rm_segment_port)**
+
+- **[hawq\_rm\_stmt\_nvseg](../../reference/guc/parameter_definitions.html#hawq_rm_stmt_nvseg)**
+
+- **[hawq\_rm\_stmt\_vseg\_memory](../../reference/guc/parameter_definitions.html#hawq_rm_stmt_vseg_memory)**
+
+- **[hawq\_rm\_tolerate\_nseg\_limit](../../reference/guc/parameter_definitions.html#hawq_rm_tolerate_nseg_limit)**
+
+- **[hawq\_rm\_yarn\_address](../../reference/guc/parameter_definitions.html#hawq_rm_yarn_address)**
+
+- **[hawq\_rm\_yarn\_app\_name](../../reference/guc/parameter_definitions.html#hawq_rm_yarn_app_name)**
+
+- **[hawq\_rm\_yarn\_queue\_name](../../reference/guc/parameter_definitions.html#hawq_rm_yarn_queue_name)**
+
+- **[hawq\_rm\_yarn\_scheduler\_address](../../reference/guc/parameter_definitions.html#hawq_rm_yarn_scheduler_address)**
+
+- **[hawq\_segment\_address\_port](../../reference/guc/parameter_definitions.html#hawq_segment_address_port)**
+
+- **[hawq\_segment\_directory](../../reference/guc/parameter_definitions.html#hawq_segment_directory)**
+
+- **[hawq\_segment\_temp\_directory](../../reference/guc/parameter_definitions.html#hawq_segment_temp_directory)**
+
+- **[integer\_datetimes](../../reference/guc/parameter_definitions.html#integer_datetimes)**
+
+- **[IntervalStyle](../../reference/guc/parameter_definitions.html#IntervalStyle)**
+
+- **[join\_collapse\_limit](../../reference/guc/parameter_definitions.html#join_collapse_limit)**
+
+- **[krb\_caseins\_users](../../reference/guc/parameter_definitions.html#krb_caseins_users)**
+
+- **[krb\_server\_keyfile](../../reference/guc/parameter_definitions.html#krb_server_keyfile)**
+
+- **[krb\_srvname](../../reference/guc/parameter_definitions.html#krb_srvname)**
+
+- **[lc\_collate](../../reference/guc/parameter_definitions.html#lc_collate)**
+
+- **[lc\_ctype](../../reference/guc/parameter_definitions.html#lc_ctype)**
+
+- **[lc\_messages](../../reference/guc/parameter_definitions.html#lc_messages)**
+
+- **[lc\_monetary](../../reference/guc/parameter_definitions.html#lc_monetary)**
+
+- **[lc\_numeric](../../reference/guc/parameter_definitions.html#lc_numeric)**
+
+- **[lc\_time](../../reference/guc/parameter_definitions.html#lc_time)**
+
+- **[listen\_addresses](../../reference/guc/parameter_definitions.html#listen_addresses)**
+
+- **[local\_preload\_libraries](../../reference/guc/parameter_definitions.html#local_preload_libraries)**
+
+- **[log\_autostats](../../reference/guc/parameter_definitions.html#log_autostats)**
+
+- **[log\_connections](../../reference/guc/parameter_definitions.html#log_connections)**
+
+- **[log\_disconnections](../../reference/guc/parameter_definitions.html#log_disconnections)**
+
+- **[log\_dispatch\_stats](../../reference/guc/parameter_definitions.html#log_dispatch_stats)**
+
+- **[log\_duration](../../reference/guc/parameter_definitions.html#log_duration)**
+
+- **[log\_error\_verbosity](../../reference/guc/parameter_definitions.html#log_error_verbosity)**
+
+- **[log\_executor\_stats](../../reference/guc/parameter_definitions.html#log_executor_stats)**
+
+- **[log\_hostname](../../reference/guc/parameter_definitions.html#log_hostname)**
+
+- **[log\_min\_duration\_statement](../../reference/guc/parameter_definitions.html#log_min_duration_statement)**
+
+- **[log\_min\_error\_statement](../../reference/guc/parameter_definitions.html#log_min_error_statement)**
+
+- **[log\_min\_messages](../../reference/guc/parameter_definitions.html#log_min_messages)**
+
+- **[log\_parser\_stats](../../reference/guc/parameter_definitions.html#log_parser_stats)**
+
+- **[log\_planner\_stats](../../reference/guc/parameter_definitions.html#log_planner_stats)**
+
+- **[log\_rotation\_age](../../reference/guc/parameter_definitions.html#log_rotation_age)**
+
+- **[log\_rotation\_size](../../reference/guc/parameter_definitions.html#log_rotation_size)**
+
+- **[log\_statement](../../reference/guc/parameter_definitions.html#log_statement)**
+
+- **[log\_statement\_stats](../../reference/guc/parameter_definitions.html#log_statement_stats)**
+
+- **[log\_timezone](../../reference/guc/parameter_definitions.html#log_timezone)**
+
+- **[log\_truncate\_on\_rotation](../../reference/guc/parameter_definitions.html#log_truncate_on_rotation)**
+
+- **[max\_appendonly\_tables](../../reference/guc/parameter_definitions.html#max_appendonly_tables)**
+
+- **[max\_connections](../../reference/guc/parameter_definitions.html#max_connections)**
+
+- **[max\_files\_per\_process](../../reference/guc/parameter_definitions.html#max_files_per_process)**
+
+- **[max\_fsm\_pages](../../reference/guc/parameter_definitions.html#max_fsm_pages)**
+
+- **[max\_fsm\_relations](../../reference/guc/parameter_definitions.html#max_fsm_relations)**
+
+- **[max\_function\_args](../../reference/guc/parameter_definitions.html#max_function_args)**
+
+- **[max\_identifier\_length](../../reference/guc/parameter_definitions.html#max_identifier_length)**
+
+- **[max\_index\_keys](../../reference/guc/parameter_definitions.html#max_index_keys)**
+
+- **[max\_locks\_per\_transaction](../../reference/guc/parameter_definitions.html#max_locks_per_transaction)**
+
+- **[max\_prepared\_transactions](../../reference/guc/parameter_definitions.html#max_prepared_transactions)**
+
+- **[max\_stack\_depth](../../reference/guc/parameter_definitions.html#max_stack_depth)**
+
+- **[optimizer](../../reference/guc/parameter_definitions.html#optimizer)**
+
+- **[optimizer\_analyze\_root\_partition](../../reference/guc/parameter_definitions.html#optimizer_analyze_root_partition)**
+
+- **[optimizer\_minidump](../../reference/guc/parameter_definitions.html#optimizer_minidump)**
+
+- **[optimizer\_parts\_to\_force\_sort\_on\_insert](../../reference/guc/parameter_definitions.html#optimizer_parts_to_force_sort_on_insert)**
+
+- **[optimizer\_prefer\_scalar\_dqa\_multistage\_agg](../../reference/guc/parameter_definitions.html#optimizer_prefer_scalar_dqa_multistage_agg)**
+
+- **[password\_encryption](../../reference/guc/parameter_definitions.html#password_encryption)**
+
+- **[pgstat\_track\_activity\_query\_size](../../reference/guc/parameter_definitions.html#pgstat_track_activity_query_size)**
+
+- **[pljava\_classpath](../../reference/guc/parameter_definitions.html#pljava_classpath)**
+
+- **[pljava\_statement\_cache\_size](../../reference/guc/parameter_definitions.html#pljava_statement_cache_size)**
+
+- **[pljava\_release\_lingering\_savepoints](../../reference/guc/parameter_definitions.html#pljava_release_lingering_savepoints)**
+
+- **[pljava\_vmoptions](../../reference/guc/parameter_definitions.html#pljava_vmoptions)**
+
+- **[port](../../reference/guc/parameter_definitions.html#port)**
+
+- **[pxf\_enable\_stat\_collection](../../reference/guc/parameter_definitions.html#pxf_enable_stat_collection)**
+
+- **[pxf\_service\_address](../../reference/guc/parameter_definitions.html#pxf_service_address)**
+
+- **[pxf\_service\_port](../../reference/guc/parameter_definitions.html#pxf_service_port)**
+
+- **[pxf\_stat\_max\_fragments](../../reference/guc/parameter_definitions.html#pxf_stat_max_fragments)**
+
+- **[random\_page\_cost](../../reference/guc/parameter_definitions.html#random_page_cost)**
+
+- **[regex\_flavor](../../reference/guc/parameter_definitions.html#regex_flavor)**
+
+- **[runaway\_detector\_activation\_percent](../../reference/guc/parameter_definitions.html#runaway_detector_activation_percent)**
+
+- **[search\_path](../../reference/guc/parameter_definitions.html#search_path)**
+
+- **[seg\_max\_connections](../../reference/guc/parameter_definitions.html#seg_max_connections)**
+
+- **[seq\_page\_cost](../../reference/guc/parameter_definitions.html#seq_page_cost)**
+
+- **[server\_encoding](../../reference/guc/parameter_definitions.html#server_encoding)**
+
+- **[server\_version](../../reference/guc/parameter_definitions.html#server_version)**
+
+- **[server\_version\_num](../../reference/guc/parameter_definitions.html#server_version_num)**
+
+- **[shared\_buffers](../../reference/guc/parameter_definitions.html#shared_buffers)**
+
+- **[shared\_preload\_libraries](../../reference/guc/parameter_definitions.html#shared_preload_libraries)**
+
+- **[ssl](../../reference/guc/parameter_definitions.html#ssl)**
+
+- **[ssl\_ciphers](../../reference/guc/parameter_definitions.html#ssl_ciphers)**
+
+- **[standard\_conforming\_strings](../../reference/guc/parameter_definitions.html#standard_conforming_strings)**
+
+- **[statement\_timeout](../../reference/guc/parameter_definitions.html#statement_timeout)**
+
+- **[superuser\_reserved\_connections](../../reference/guc/parameter_definitions.html#superuser_reserved_connections)**
+
+- **[tcp\_keepalives\_count](../../reference/guc/parameter_definitions.html#tcp_keepalives_count)**
+
+- **[tcp\_keepalives\_idle](../../reference/guc/parameter_definitions.html#tcp_keepalives_idle)**
+
+- **[tcp\_keepalives\_interval](../../reference/guc/parameter_definitions.html#tcp_keepalives_interval)**
+
+- **[temp\_buffers](../../reference/guc/parameter_definitions.html#temp_buffers)**
+
+- **[TimeZone](../../reference/guc/parameter_definitions.html#TimeZone)**
+
+- **[timezone\_abbreviations](../../reference/guc/parameter_definitions.html#timezone_abbreviations)**
+
+- **[track\_activities](../../reference/guc/parameter_definitions.html#track_activities)**
+
+- **[track\_counts](../../reference/guc/parameter_definitions.html#track_counts)**
+
+- **[transaction\_isolation](../../reference/guc/parameter_definitions.html#transaction_isolation)**
+
+- **[transaction\_read\_only](../../reference/guc/parameter_definitions.html#transaction_read_only)**
+
+- **[transform\_null\_equals](../../reference/guc/parameter_definitions.html#transform_null_equals)**
+
+- **[unix\_socket\_directory](../../reference/guc/parameter_definitions.html#unix_socket_directory)**
+
+- **[unix\_socket\_group](../../reference/guc/parameter_definitions.html#unix_socket_group)**
+
+- **[unix\_socket\_permissions](../../reference/guc/parameter_definitions.html#unix_socket_permissions)**
+
+- **[update\_process\_title](../../reference/guc/parameter_definitions.html#update_process_title)**
+
+- **[vacuum\_cost\_delay](../../reference/guc/parameter_definitions.html#vacuum_cost_delay)**
+
+- **[vacuum\_cost\_limit](../../reference/guc/parameter_definitions.html#vacuum_cost_limit)**
+
+- **[vacuum\_cost\_page\_dirty](../../reference/guc/parameter_definitions.html#vacuum_cost_page_dirty)**
+
+- **[vacuum\_cost\_page\_miss](../../reference/guc/parameter_definitions.html#vacuum_cost_page_miss)**
+
+- **[vacuum\_freeze\_min\_age](../../reference/guc/parameter_definitions.html#vacuum_freeze_min_age)**
+
+- **[xid\_stop\_limit](../../reference/guc/parameter_definitions.html#xid_stop_limit)**
+
+
+
+## add\_missing\_from <a name="add_missing_from"></a>
+
+Automatically adds missing table references to FROM clauses. Present for compatibility with releases of PostgreSQL prior to 8.1, where this behavior was allowed by default.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | off | master, session, reload |
+
+
+## application\_name <a name="application_name"></a>
+
+Sets the application name for a client session. For example, if connecting via `psql`, this will be set to `psql`. Setting an application name allows it to be reported in log messages and statistics views.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| string | | master, session, reload |
+
+
+
+## array\_nulls <a name="array_nulls"></a>
+
+This controls whether the array input parser recognizes unquoted NULL as specifying a null array element. By default, this is on, allowing array values containing null values to be entered.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, session, reload |
+
+
+
+## authentication\_timeout <a name="authentication_timeout"></a>
+
+Maximum time to complete client authentication. This prevents hung clients from occupying a connection indefinitely.
+
+| Value Range | Default | Set Classifications |
+|---------------------------------------------|---------|------------------------|
+| Any valid time expression (number and unit) | 1min | local, system, restart |
+
+
+## backslash\_quote <a name="backslash_quote"></a>
+
+This controls whether a quote mark can be represented by `\'` in a string literal. The preferred, SQL-standard way to represent a quote mark is by doubling it (`''`) but PostgreSQL has historically also accepted `\'`. However, use of `\'` creates security risks because in some client character set encodings, there are multibyte characters in which the last byte is numerically equivalent to ASCII `\`.
+
+<table>
+<colgroup>
+<col width="33%" />
+<col width="33%" />
+<col width="33%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Value Range</th>
+<th>Default</th>
+<th>Set Classifications</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>on (allow <code class="ph codeph">\'</code> always)
+<p>off (reject always)</p>
+<p>safe_encoding (allow only if client encoding does not allow ASCII <code class="ph codeph">\</code> within a multibyte character)</p></td>
+<td>safe_encoding</td>
+<td>master, session, reload</td>
+</tr>
+</tbody>
+</table>
+
+
+## block\_size <a name="block_size"></a>
+
+Reports the size of a disk block.
+
+| Value Range | Default | Set Classifications |
+|-----------------|---------|---------------------|
+| number of bytes | 32768 | read only |
+
+
+## bonjour\_name <a name="bonjour_name"></a>
+
+Specifies the Bonjour broadcast name. By default, the computer name is used, specified as an empty string. This option is ignored if the server was not compiled with Bonjour support.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| string | unset | master, system, restart |
+
+
+## check\_function\_bodies <a name="check_function_bodies"></a>
+
+When set to off, disables validation of the function body string during `CREATE FUNCTION`. Disabling validation is occasionally useful to avoid problems such as forward references when restoring function definitions from a dump.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, session, reload |
+
+
+## client\_encoding <a name="client_encoding"></a>
+
+Sets the client-side encoding (character set). The default is to use the same as the database encoding. See [Supported Character Sets](http://www.postgresql.org/docs/8.1/static/multibyte.html#MULTIBYTE-CHARSET-SUPPORTED) in the PostgreSQL documentation.
+
+| Value Range | Default | Set Classifications |
+|---------------|---------|-------------------------|
+| character set | UTF8 | master, session, reload |
+
+
+## client\_min\_messages <a name="client_min_messages"></a>
+
+Controls which message levels are sent to the client. Each level includes all the levels that follow it. The later the level, the fewer messages are sent.
+
+<table>
+<colgroup>
+<col width="33%" />
+<col width="33%" />
+<col width="33%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Value Range</th>
+<th>Default</th>
+<th>Set Classifications</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>DEBUG5
+<p>DEBUG4</p>
+<p>DEBUG3</p>
+<p>DEBUG2</p>
+<p>DEBUG1</p>
+<p>LOG NOTICE</p>
+<p>WARNING</p>
+<p>ERROR</p>
+<p>FATAL</p>
+<p>PANIC</p></td>
+<td>NOTICE</td>
+<td>master, session, reload</td>
+</tr>
+</tbody>
+</table>
+
+## cpu\_index\_tuple\_cost <a name="cpu_index_tuple_cost"></a>
+
+For the legacy query optimizer (planner), sets the estimate of the cost of processing each index row during an index scan. This is measured as a fraction of the cost of a sequential page fetch.
+
+| Value Range | Default | Set Classifications |
+|----------------|---------|-------------------------|
+| floating point | 0.005 | master, session, reload |
+
+
+
+## cpu\_operator\_cost <a name="cpu_operator_cost"></a>
+
+For the legacy query optimizer (planner), sets the estimate of the cost of processing each operator in a WHERE clause. This is measured as a fraction of the cost of a sequential page fetch.
+
+| Value Range | Default | Set Classifications |
+|----------------|---------|-------------------------|
+| floating point | 0.0025 | master, session, reload |
+
+
+## cpu\_tuple\_cost <a name="cpu_tuple_cost"></a>
+
+For the legacy query optimizer (planner), Sets the estimate of the cost of processing each row during a query. This is measured as a fraction of the cost of a sequential page fetch.
+
+| Value Range | Default | Set Classifications |
+|----------------|---------|-------------------------|
+| floating point | 0.01 | master, session, reload |
+
+
+## cursor\_tuple\_fraction <a name="cursor_tuple_fraction"></a>
+
+Tells the legacy query optimizer (planner) how many rows are expected to be fetched in a cursor query, thereby allowing the legacy optimizer to use this information to optimize the query plan. The default of 1 means all rows will be fetched.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| integer | 1 | master, session, reload |
+
+
+## custom\_variable\_classes <a name="custom_variable_classes"></a>
+
+Specifies one or several class names to be used for custom variables. A custom variable is a variable not normally known to the server but used by some add-on modules. Such variables must have names consisting of a class name, a dot, and a variable name.
+
+| Value Range | Default | Set Classifications |
+|-------------------------------------|---------|------------------------|
+| comma-separated list of class names | unset | local, system, restart |
+
+
+## DateStyle <a name="DateStyle"></a>
+
+Sets the display format for date and time values, as well as the rules for interpreting ambiguous date input values. This variable contains two independent components: the output format specification and the input/output specification for year/month/day ordering.
+
+<table>
+<colgroup>
+<col width="33%" />
+<col width="33%" />
+<col width="33%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Value Range</th>
+<th>Default</th>
+<th>Set Classifications</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td><format>, <date style>
+<p>where:</p>
+<p><format> is ISO, Postgres, SQL, or German</p>
+<p><date style> is DMY, MDY, or YMD</p></td>
+<td>ISO, MDY</td>
+<td>master, session, reload</td>
+</tr>
+</tbody>
+</table>
+
+## db\_user\_namespace <a name="db_user_namespace"></a>
+
+This enables per-database user names. If on, you should create users as *username@dbname*. To create ordinary global users, simply append @ when specifying the user name in the client.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|------------------------|
+| Boolean | off | local, system, restart |
+
+
+## deadlock\_timeout <a name="deadlock_timeout"></a>
+
+The time, in milliseconds, to wait on a lock before checking to see if there is a deadlock condition. On a heavily loaded server you might want to raise this value. Ideally the setting should exceed your typical transaction time, so as to improve the odds that a lock will be released before the waiter decides to check for deadlock.
+
+| Value Range | Default | Set Classifications |
+|------------------------|---------|------------------------|
+| integer (milliseconds) | 1000 | local, system, restart |
+
+
+## debug\_assertions <a name="debug_assertions"></a>
+
+Turns on various assertion checks.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|------------------------|
+| Boolean | off | local, system, restart |
+
+
+## debug\_pretty\_print <a name="debug_pretty_print"></a>
+
+Indents debug output to produce a more readable but much longer output format. *client\_min\_messages* or *log\_min\_messages* must be DEBUG1 or lower.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | off | master, session, reload |
+
+
+## debug\_print\_parse <a name="debug_print_parse"></a>
+
+
+For each executed query, prints the resulting parse tree. *client\_min\_messages* or *log\_min\_messages* must be DEBUG1 or lower.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | off | master, session, reload |
+
+
+## debug\_print\_plan <a name="debug_print_plan"></a>
+
+For each executed query, prints the HAWQ parallel query execution plan. *client\_min\_messages* or *log\_min\_messages* must be DEBUG1 or lower.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | off | master, session, reload |
+
+## debug\_print\_prelim\_plan <a name="debug_print_prelim_plan"></a>
+
+For each executed query, prints the preliminary query plan. *client\_min\_messages* or *log\_min\_messages* must be DEBUG1 or lower.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | off | master, session, reload |
+
+
+## debug\_print\_rewritten <a name="debug_print_rewritten"></a>
+
+For each executed query, prints the query rewriter output. *client\_min\_messages* or *log\_min\_messages* must be DEBUG1 or lower.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | off | master, session, reload |
+
+
+## debug\_print\_slice\_table <a name="debug_print_slice_table"></a>
+
+For each executed query, prints the HAWQ query slice plan. *client\_min\_messages* or *log\_min\_messages* must be DEBUG1 or lower.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | off | master, session, reload |
+
+## default\_hash\_table\_bucket\_number <a name="topic_fqj_4fd_kv"></a>
+
+The default number of hash buckets to use when executing a query statement on a hash table. Due to dynamic allocation, when the query is actually executed, the number of virtual segments may differ from this number depending on the query's needs. The total number of segments should never exceed the maximum set in `hawq_rm_nvseg_perquery_limit`.
+
+When expanding the cluster, you should adjust this number to reflect the number of nodes in the new cluster times the number of virtual segments per node. See [Expanding a Cluster](../../admin/ClusterExpansion.html) and [Creating and Managing Tables](../../ddl/ddl-table.html) for more details on modifying this parameter.
+
+| Value Range | Default | Set Classifications |
+|----------------|-----------------|-------------------------|
+| integer > 0 | 6\*SegmentCount | master, session, reload |
+
+
+## default\_statistics\_target <a name="default_statistics_target"></a>
+
+Sets the default statistics target for table columns that have not had a column-specific target set via `ALTER TABLE SET STATISTICS`. Larger values increase the time needed to do `ANALYZE`, but may improve the quality of the legacy query optimizer (planner) estimates.
+
+| Value Range | Default | Set Classifications |
+|----------------|---------|-------------------------|
+| integer > 0 | 25 | master, session, reload |
+
+
+## default\_tablespace <a name="default_tablespace"></a>
+
+The default tablespace in which to create objects (tables and indexes) when a `CREATE` command does not explicitly specify a tablespace.
+
+| Value Range | Default | Set Classifications |
+|----------------------|---------|-------------------------|
+| name of a tablespace | unset | master, session, reload |
+
+
+## default\_transaction\_isolation <a name="default_transaction_isolation"></a>
+
+Controls the default isolation level of each new transaction.
+
+<table>
+<colgroup>
+<col width="33%" />
+<col width="33%" />
+<col width="33%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Value Range</th>
+<th>Default</th>
+<th>Set Classifications</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>read committed
+<p>read uncommitted</p>
+<p>repeatable read</p>
+<p>serializable</p></td>
+<td>read committed</td>
+<td>master, session, reload</td>
+</tr>
+</tbody>
+</table>
+
+## default\_transaction\_read\_only <a name="default_transaction_read_only"></a>
+
+Controls the default read-only status of each new transaction. A read-only SQL transaction cannot alter non-temporary tables.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | off | master, session, reload |
+
+
+
+
+## dfs\_url <a name="dfs_url"></a>
+
+See [hawq\_dfs\_url](#hawq_dfs_url).
+
+
+## dynamic\_library\_path <a name="dynamic_library_path"></a>
+
+If a dynamically loadable module needs to be opened and the file name specified in the `CREATE FUNCTION` or `LOAD` command does not have a directory component (i.e. the name does not contain a slash), the system will search this path for the required file. The compiled-in PostgreSQL package library directory is substituted for $libdir. This is where the modules provided by the standard PostgreSQL distribution are installed.
+
+| Value Range | Default | Set Classifications |
+|--------------------------------------------------------|---------|-------------------------|
+| a list of absolute directory paths separated by colons | $libdir | master, session, reload |
+
+## effective\_cache\_size <a name="effective_cache_size"></a>
+
+Sets the assumption about the effective size of the disk cache that is available to a single query for the legacy query optimizer (planner). This is factored into estimates of the cost of using an index; a higher value makes it more likely index scans will be used, a lower value makes it more likely sequential scans will be used. This parameter has no effect on the size of shared memory allocated by a HAWQ server instance, nor does it reserve kernel disk cache; it is used only for estimation purposes.
+
+| Value Range | Default | Set Classifications |
+|----------------|---------|-------------------------|
+| floating point | 512MB | master, session, reload |
+
+
+## enable\_bitmapscan <a name="enable_bitmapscan"></a>
+
+Enables or disables the use of bitmap-scan plan types by the legacy query optimizer (planner). Note that this is different than a Bitmap Index Scan. A Bitmap Scan means that indexes will be dynamically converted to bitmaps in memory when appropriate, giving faster index performance on complex queries against very large tables. It is used when there are multiple predicates on different indexed columns. Each bitmap per column can be compared to create a final list of selected tuples.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, session, reload |
+
+
+## enable\_groupagg <a name="enable_groupagg"></a>
+
+Enables or disables the use of group aggregation plan types by the legacy query optimizer (planner).
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, session, reload |
+
+## enable\_hashagg <a name="enable_hashagg"></a>
+
+Enables or disables the use of hash aggregation plan types by the legacy query optimizer (planner).
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, session, reload |
+
+
+
+## enable\_hashjoin <a name="enable_hashjoin"></a>
+
+Enables or disables the use of hash-join plan types by the legacy query optimizer (planner).
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, session, reload |
+
+## enable\_indexscan <a name="enable_indexscan"></a>
+
+Enables or disables the use of index-scan plan types by the legacy query optimizer (planner).
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, session, reload |
+
+## enable\_mergejoin <a name="enable_mergejoin"></a>
+
+Enables or disables the use of merge-join plan types by the legacy query optimizer (planner). Merge join is based on the idea of sorting the left- and right-hand tables into order and then scanning them in parallel. So, both data types must be capable of being fully ordered, and the join operator must be one that can only succeed for pairs of values that fall at the 'same place' in the sort order. In practice this means that the join operator must behave like equality.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | off | master, session, reload |
+
+## enable\_nestloop <a name="enable_nestloop"></a>
+
+Enables or disables the use of nested-loop join plans by the legacy query optimizer (planner). It's not possible to suppress nested-loop joins entirely, but turning this variable off discourages the legacy optimizer from using one if there are other methods available.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | off | master, session, reload |
+
+## enable\_seqscan<a name="enable_seqscan"></a>
+
+Enables or disables the use of sequential scan plan types by the legacy query optimizer (planner). It's not possible to suppress sequential scans entirely, but turning this variable off discourages the legacy optimizer from using one if there are other methods available.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, session, reload |
+
+## enable\_sort<a name="enable_sort"></a>
+
+Enables or disables the use of explicit sort steps by the legacy query optimizer (planner). It's not possible to suppress explicit sorts entirely, but turning this variable off discourages the legacy optimizer from using one if there are other methods available.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, session, reload |
+
+## enable\_tidscan<a name="enable_tidscan"></a>
+
+Enables or disables the use of tuple identifier (TID) scan plan types by the legacy query optimizer (planner).
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, session, reload |
+
+## escape\_string\_warning<a name="escape_string_warning"></a>
+
+When on, a warning is issued if a backslash (\\) appears in an ordinary string literal ('...' syntax). Escape string syntax (E'...') should be used for escapes, because in future versions, ordinary strings will have the SQL standard-conforming behavior of treating backslashes literally.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, session, reload |
+
+## explain\_pretty\_print<a name="explain_pretty_print"></a>
+
+Determines whether EXPLAIN VERBOSE uses the indented or non-indented format for displaying detailed query-tree dumps.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, session, reload |
+
+## extra\_float\_digits<a name="extra_float_digits"></a>
+
+Adjusts the number of digits displayed for floating-point values, including float4, float8, and geometric data types. The parameter value is added to the standard number of digits. The value can be set as high as 2, to include partially-significant digits; this is especially useful for dumping float data that needs to be restored exactly. Or it can be set negative to suppress unwanted digits.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| integer | 0 | master, session, reload |
+
+## from\_collapse\_limit<a name="from_collapse_limit"></a>
+
+The legacy query optimizer (planner) will merge sub-queries into upper queries if the resulting FROM list would have no more than this many items. Smaller values reduce planning time but may yield inferior query plans.
+
+| Value Range | Default | Set Classifications |
+|-----------------|---------|-------------------------|
+| integer (1-*n*) | 20 | master, session, reload |
+
+## gp\_adjust\_selectivity\_for\_outerjoins<a name="gp_adjust_selectivity_for_outerjoins"></a>
+
+Enables the selectivity of NULL tests over outer joins.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, session, reload |
+
+## gp\_analyze\_relative\_error<a name="gp_analyze_relative_error"></a>
+
+Sets the estimated acceptable error in the cardinality of the table " a value of 0.5 is supposed to be equivalent to an acceptable error of 50% (this is the default value used in PostgreSQL). If the statistics collected during `ANALYZE` are not producing good estimates of cardinality for a particular table attribute, decreasing the relative error fraction (accepting less error) tells the system to sample more rows.
+
+| Value Range | Default | Set Classifications |
+|-------------------------|---------|-------------------------|
+| floating point < 1.0 | 0.25 | master, session, reload |
+
+## gp\_autostats\_mode<a name="gp_autostats_mode"></a>
+
+Specifies the mode for triggering automatic statistics collection with `ANALYZE`. The `on_no_stats` option triggers statistics collection for `CREATE TABLE AS SELECT`, `INSERT`, or `COPY` operations on any table that has no existing statistics.
+
+**Warning:** Depending on the specific nature of your database operations, automatic statistics collection can have a negative performance impact. Carefully evaluate whether the default value of `on_no_stats` is appropriate for your system.
+
+The `on_change` option triggers statistics collection only when the number of rows affected exceeds the threshold defined by `gp_autostats_on_change_threshold`. Operations that can trigger automatic statistics collection with `on_change` are:
+
+`CREATE TABLE AS SELECT`
+
+`UPDATE`
+
+`DELETE`
+
+`INSERT`
+
+`COPY`
+
+Default is `on_no_stats`.
+
+**Note:** For partitioned tables, automatic statistics collection is not triggered if data is inserted from the top-level parent table of a partitioned table.
+Automatic statistics collection is triggered if data is inserted directly in a leaf table (where the data is stored) of the partitioned table. Statistics are collected only on the leaf table.
+
+<table>
+<colgroup>
+<col width="33%" />
+<col width="33%" />
+<col width="33%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Value Range</th>
+<th>Default</th>
+<th>Set Classifications</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>none
+<p>on_change</p>
+<p>on_no_stats</p></td>
+<td>on_no_ stats</td>
+<td>master, session, reload</td>
+</tr>
+</tbody>
+</table>
+
+## gp\_autostats\_on\_change\_threshhold<a name="topic_imj_zhf_gw"></a>
+
+Specifies the threshold for automatic statistics collection when `gp_autostats_mode` is set to `on_change`. When a triggering table operation affects a number of rows exceeding this threshold, `ANALYZE `is added and statistics are collected for the table.
+
+| Value Range | Default | Set Classifications |
+|-------------|------------|-------------------------|
+| integer | 2147483647 | master, session, reload |
+
+## gp\_backup\_directIO<a name="gp_backup_directIO"></a>
+
+Direct I/O allows HAWQ to bypass the buffering of memory within the file system cache for backup. When Direct I/O is used for a file, data is transferred directly from the disk to the application buffer, without the use of the file buffer cache.
+
+Direct I/O is supported only on Red Hat Enterprise Linux, CentOS, and SUSE.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| on, off | off | master, session, reload |
+
+## gp\_backup\_directIO\_read\_chunk\_mb<a name="gp_backup_directIO_read_chunk_mb"></a>
+
+Sets the chunk size in MB when Direct I/O is enabled with [gp\_backup\_directIO](#gp_backup_directIO). The default chunk size is 20MB.
+
+The default value is the optimal setting. Decreasing it will increase the backup time and increasing it will result in little change to backup time.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|------------------------|
+| 1-200 | 20 MB | local, session, reload |
+
+## gp\_cached\_segworkers\_threshold<a name="gp_cached_segworkers_threshold"></a>
+
+When a user starts a session with HAWQ and issues a query, the system creates groups or 'gangs' of worker processes on each segment to do the work. After the work is done, the segment worker processes are destroyed except for a cached number which is set by this parameter. A lower setting conserves system resources on the segment hosts, but a higher setting may improve performance for power-users that want to issue many complex queries in a row.
+
+| Value Range | Default | Set Classifications |
+|----------------|---------|-------------------------|
+| integer > 0 | 5 | master, session, reload |
+
+## gp\_command\_count<a name="gp_command_count"></a>
+
+Shows how many commands the master has received from the client. Note that a single SQL command might actually involve more than one command internally, so the counter may increment by more than one for a single query. This counter also is shared by all of the segment processes working on the command.
+
+| Value Range | Default | Set Classifications |
+|----------------|---------|---------------------|
+| integer > 0 | 1 | read only |
+
+## gp\_connections\_per\_thread<a name="gp_connections_per_thread"></a>
+
+A value larger than or equal to the number of segments means that each slice in a query plan will get its own thread when dispatching to the segments. A value of 0 indicates that the dispatcher should use a single thread when dispatching all query plan slices to a segment. Lower values will use more threads, which utilizes more resources on the master. Typically, the default does not need to be changed unless there is a known throughput performance problem.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| integer | 64 | master, session, reload |
+
+## gp\_debug\_linger<a name="gp_debug_linger"></a>
+
+Number of seconds for a HAWQ process to linger after a fatal internal error.
+
+| Value Range | Default | Set Classifications |
+|---------------------------------------------|---------|-------------------------|
+| Any valid time expression (number and unit) | 0 | master, session, reload |
+
+## gp\_dynamic\_partition\_pruning<a name="gp_dynamic_partition_pruning"></a>
+
+Enables plans that can dynamically eliminate the scanning of partitions.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| on/off | on | master, session, reload |
+
+## gp\_enable\_agg\_distinct<a name="gp_enable_agg_distinct"></a>
+
+Enables or disables two-phase aggregation to compute a single distinct-qualified aggregate. This applies only to subqueries that include a single distinct-qualified aggregate function.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, session, reload |
+
+## gp\_enable\_agg\_distinct\_pruning<a name="gp_enable_agg_distinct_pruning"></a>
+
+Enables or disables three-phase aggregation and join to compute distinct-qualified aggregates. This applies only to subqueries that include one or more distinct-qualified aggregate functions.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, session, reload |
+
+## gp\_enable\_direct\_dispatch<a name="gp_enable_direct_dispatch"></a>
+
+Enables or disables the dispatching of targeted query plans for queries that access data on a single segment. When on, queries that target rows on a single segment will only have their query plan dispatched to that segment (rather than to all segments). This significantly reduces the response time of qualifying queries as there is no interconnect setup involved. Direct dispatch does require more CPU utilization on the master.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, system, restart |
+
+## gp\_enable\_fallback\_plan<a name="gp_enable_fallback_plan"></a>
+
+Allows use of disabled plan types when a query would not be feasible without them.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, session, reload |
+
+## gp\_enable\_fast\_sri<a name="gp_enable_fast_sri"></a>
+
+When set to `on`, the legacy query optimizer (planner) plans single row inserts so that they are sent directly to the correct segment instance (no motion operation required). This significantly improves performance of single-row-insert statements.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, session, reload |
+
+## gp\_enable\_groupext\_distinct\_gather<a name="gp_enable_groupext_distinct_gather"></a>
+
+Enables or disables gathering data to a single node to compute distinct-qualified aggregates on grouping extension queries. When this parameter and `gp_enable_groupext_distinct_pruning` are both enabled, the legacy query optimizer (planner) uses the cheaper plan.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, session, reload |
+
+## gp\_enable\_groupext\_distinct\_pruning<a name="gp_enable_groupext_distinct_pruning"></a>
+
+Enables or disables three-phase aggregation and join to compute distinct-qualified aggregates on grouping extension queries. Usually, enabling this parameter generates a cheaper query plan that the legacy query optimizer (planner) will use in preference to existing plan.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, session, reload |
+
+## gp\_enable\_multiphase\_agg<a name="gp_enable_multiphase_agg"></a>
+
+Enables or disables the use of two or three-stage parallel aggregation plans legacy query optimizer (planner). This approach applies to any subquery with aggregation. If `gp_enable_multiphase_agg` is off, then` gp_enable_agg_distinct` and `gp_enable_agg_distinct_pruning` are disabled.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, session, reload |
+
+## gp\_enable\_predicate\_propagation<a name="gp_enable_predicate_propagation"></a>
+
+When enabled, the legacy query optimizer (planner) applies query predicates to both table expressions in cases where the tables are joined on their distribution key column(s). Filtering both tables prior to doing the join (when possible) is more efficient.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, session, reload |
+
+## gp\_enable\_preunique<a name="gp_enable_preunique"></a>
+
+Enables two-phase duplicate removal for `SELECT DISTINCT` queries (not `SELECT COUNT(DISTINCT)`). When enabled, it adds an extra `SORT DISTINCT` set of plan nodes before motioning. In cases where the distinct operation greatly reduces the number of rows, this extra `SORT DISTINCT` is much cheaper than the cost of sending the rows across the Interconnect.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, session, reload |
+
+## gp\_enable\_sequential\_window\_plans<a name="gp_enable_sequential_window_plans"></a>
+
+If on, enables non-parallel (sequential) query plans for queries containing window function calls. If off, evaluates compatible window functions in parallel and rejoins the results. This is an experimental parameter.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, session, reload |
+
+## gp\_enable\_sort\_distinct<a name="gp_enable_sort_distinct"></a>
+
+Enable duplicates to be removed while sorting.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, session, reload |
+
+## gp\_enable\_sort\_limit<a name="gp_enable_sort_limit"></a>
+
+Enable `LIMIT` operation to be performed while sorting. Sorts more efficiently when the plan requires the first *limit\_number* of rows at most.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, session, reload |
+
+## gp\_external\_enable\_exec<a name="gp_external_enable_exec"></a>
+
+Enables or disables the use of external tables that execute OS commands or scripts on the segment hosts (`CREATE EXTERNAL TABLE EXECUTE` syntax).
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, system, restart |
+
+## gp\_external\_grant\_privileges<a name="gp_external_grant_privileges"></a>
+
+Enables or disables non-superusers to issue a `CREATE EXTERNAL [WEB] TABLE` command in cases where the `LOCATION` clause specifies specifies `http` or `gpfdist`. The ability to create an external table can be granted to a role using `CREATE ROLE` or `ALTER ROLE`.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | off | master, system, restart |
+
+## gp\_external\_max\_segs<a name="gp_external_max_segs"></a>
+
+Sets the number of segments that will scan external table data during an external table operation, the purpose being not to overload the system with scanning data and take away resources from other concurrent operations. This only applies to external tables that use the `gpfdist://` protocol to access external table data.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| integer | 64 | master, session, reload |
+
+## gp\_filerep\_tcp\_keepalives\_count<a name="gp_filerep_tcp_keepalives_count"></a>
+
+How many keepalives may be lost before the connection is considered dead. A value of 0 uses the system default. If TCP\_KEEPCNT is not supported, this parameter must be 0.
+
+| Value Range | Default | Set Classifications |
+|---------------------------|---------|------------------------|
+| number of lost keepalives | 2 | local, system, restart |
+
+## gp\_filerep\_tcp\_keepalives\_idle<a name="gp_filerep_tcp_keepalives_idle"></a>
+
+Number of seconds between sending keepalives on an otherwise idle connection. A value of 0 uses the system default. If TCP\_KEEPIDLE is not supported, this parameter must be 0.
+
+| Value Range | Default | Set Classifications |
+|-------------------|---------|------------------------|
+| number of seconds | 1 min | local, system, restart |
+
+## gp\_filerep\_tcp\_keepalives\_interval<a name="gp_filerep_tcp_keepalives_interval"></a>
+
+How many seconds to wait for a response to a keepalive before retransmitting. A value of 0 uses the system default. If TCP\_KEEPINTVL is not supported, this parameter must be 0.
+
+| Value Range | Default | Set Classifications |
+|-------------------|---------|------------------------|
+| number of seconds | 30 sec | local, system, restart |
+
+## gp\_hashjoin\_tuples\_per\_bucket<a name="gp_hashjoin_tuples_per_bucket"></a>
+
+Sets the target density of the hash table used by HashJoin operations. A smaller value will tend to produce larger hash tables, which can increase join performance.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| integer | 5 | master, session, reload |
+
+## gp\_idf\_deduplicate<a name="gp_idf_deduplicate"></a>
+
+Changes the strategy to compute and process MEDIAN, and PERCENTILE\_DISC.
+
+<table>
+<colgroup>
+<col width="33%" />
+<col width="33%" />
+<col width="33%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Value Range</th>
+<th>Default</th>
+<th>Set Classifications</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>auto
+<p>none</p>
+<p>force</p></td>
+<td>auto</td>
+<td>master, session, reload</td>
+</tr>
+</tbody>
+</table>
+
+## gp\_interconnect\_fc\_method<a name="gp_interconnect_fc_method"></a>
+
+Specifies the flow control method used for UDP interconnect when the value of [gp\_interconnect\_type](#gp_interconnect_type) is UDPIFC.
+
+For capacity based flow control, senders do not send packets when receivers do not have the capacity.
+
+Loss based flow control is based on capacity based flow control, and also tunes the sending speed according to packet losses.
+
+<table>
+<colgroup>
+<col width="33%" />
+<col width="33%" />
+<col width="33%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Value Range</th>
+<th>Default</th>
+<th>Set Classifications</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>CAPACITY
+<p>LOSS</p></td>
+<td>LOSS</td>
+<td>master, session, reload</td>
+</tr>
+</tbody>
+</table>
+
+## gp\_interconnect\_hash\_multiplier<a name="gp_interconnect_hash_multiplier"></a>
+
+Sets the size of the hash table used by the UDP interconnect to track connections. This number is multiplied by the number of segments to determine the number of buckets in the hash table. Increasing the value may increase interconnect performance for complex multi-slice queries (while consuming slightly more memory on the segment hosts).
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| 2-25 | 2 | master, session, reload |
+
+## gp\_interconnect\_queue\_depth<a name="gp_interconnect_queue_depth"></a>
+
+Sets the amount of data per-peer to be queued by the UDP interconnect on receivers (when data is received but no space is available to receive it the data will be dropped, and the transmitter will need to resend it). Increasing the depth from its default value will cause the system to use more memory; but may increase performance. It is reasonable for this to be set between 1 and 10. Queries with data skew potentially perform better when this is increased. Increasing this may radically increase the amount of memory used by the system.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| 1-2048 | 4 | master, session, reload |
+
+## gp\_interconnect\_setup\_timeout<a name="gp_interconnect_setup_timeout"></a>
+
+When the interconnect type is UDP, the time to wait for the Interconnect to complete setup before it times out.
+
+This parameter is used only when [gp\_interconnect\_type](#gp_interconnect_type) is set to UDP.
+
+| Value Range | Default | Set Classifications |
+|---------------------------------------------|---------|-------------------------|
+| Any valid time expression (number and unit) | 2 hours | master, session, reload |
+
+## gp\_interconnect\_snd\_queue\_depth<a name="gp_interconnect_snd_queue_depth"></a>
+
+Sets the amount of data per-peer to be queued by the UDP interconnect on senders. Increasing the depth from its default value will cause the system to use more memory; but may increase performance. Reasonable values for this parameter are between 1 and 4. Increasing the value might radically increase the amount of memory used by the system.
+
+This parameter is used only when [gp\_interconnect\_type](#gp_interconnect_type) is set to UDPIFC.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| 1 - 4096 | 2 | master, session, reload |
+
+## gp\_interconnect\_type<a name="gp_interconnect_type"></a>
+
+Sets the networking protocol used for Interconnect traffic. With the TCP protocol, HAWQ has an upper limit of 1000 segment instances - less than that if the query workload involves complex, multi-slice queries.
+
+UDP allows for greater interconnect scalability. Note that the HAWQ software does the additional packet verification and checking not performed by UDP, so reliability and performance is equivalent to TCP.
+
+UDPIFC specifies using UDP with flow control for interconnect traffic. Specify the interconnect flow control method with [gp\_interconnect\_fc\_method](#gp_interconnect_fc_method).
+
+<table>
+<colgroup>
+<col width="33%" />
+<col width="33%" />
+<col width="33%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Value Range</th>
+<th>Default</th>
+<th>Set Classifications</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>TCP
+<p>UDP</p>
+<p>UDPIFC</p></td>
+<td>UDPIFC</td>
+<td>local, system, restart</td>
+</tr>
+</tbody>
+</table>
+
+## gp\_log\_format<a name="gp_log_format"></a>
+
+Specifies the format of the server log files. If using *hawq\_toolkit* administrative schema, the log files must be in CSV format.
+
+<table>
+<colgroup>
+<col width="33%" />
+<col width="33%" />
+<col width="33%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Value Range</th>
+<th>Default</th>
+<th>Set Classifications</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>csv
+<p>text</p></td>
+<td>csv</td>
+<td>local, system, restart</td>
+</tr>
+</tbody>
+</table>
+
+## gp\_max\_csv\_line\_length<a name="gp_max_csv_line_length"></a>
+
+The maximum length of a line in a CSV formatted file that will be imported into the system. The default is 1MB (1048576 bytes). Maximum allowed is 4MB (4194184 bytes). The default may need to be increased if using the *hawq\_toolkit* administrative schema to read HAWQ log files.
+
+| Value Range | Default | Set Classifications |
+|-----------------|---------|------------------------|
+| number of bytes | 1048576 | local, system, restart |
+
+## gp\_max\_databases<a name="gp_max_databases"></a>
+
+The maximum number of databases allowed in a HAWQ system.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| integer | 16 | master, system, restart |
+
+## gp\_max\_filespaces<a name="gp_max_filespaces"></a>
+
+The maximum number of filespaces allowed in a HAWQ system.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| integer | 8 | master, system, restart |
+
+## gp\_max\_packet\_size<a name="gp_max_packet_size"></a>
+
+Sets the size (in bytes) of messages sent by the UDP interconnect, and sets the tuple-serialization chunk size for both the UDP and TCP interconnect.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| 512-65536 | 8192 | master, system, restart |
+
+## gp\_max\_plan\_size<a name="gp_max_plan_size"></a>
+
+Specifies the total maximum uncompressed size of a query execution plan multiplied by the number of Motion operators (slices) in the plan. If the size of the query plan exceeds the value, the query is cancelled and an error is returned. A value of 0 means that the size of the plan is not monitored.
+
+You can specify a value in KB,MB, or GB. The default unit is KB. For example, a value of 200 is 200KB. A value of 1GB is the same as 1024MB or 1048576KB.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|----------------------------|
+| integer | 0 | master, superuser, session |
+
+## gp\_max\_tablespaces<a name="gp_max_tablespaces"></a>
+
+The maximum number of tablespaces allowed in a HAWQ system.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| integer | 16 | master, system, restart |
+
+## gp\_motion\_cost\_per\_row<a name="gp_motion_cost_per_row"></a>
+
+Sets the legacy query optimizer (planner) cost estimate for a Motion operator to transfer a row from one segment to another, measured as a fraction of the cost of a sequential page fetch. If 0, then the value used is two times the value of *cpu\_tuple\_cost*.
+
+| Value Range | Default | Set Classifications |
+|----------------|---------|-------------------------|
+| floating point | 0 | master, session, reload |
+
+## gp\_reject\_percent\_threshold<a name="gp_reject_percent_threshold"></a>
+
+For single row error handling on COPY and external table SELECTs, sets the number of rows processed before SEGMENT REJECT LIMIT *n* PERCENT starts calculating.
+
+| Value Range | Default | Set Classifications |
+|-----------------|---------|-------------------------|
+| integer (1-*n*) | 300 | master, session, reload |
+
+## gp\_reraise\_signal<a name="gp_reraise_signal"></a>
+
+If enabled, will attempt to dump core if a fatal server error occurs.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, session, reload |
+
+## gp\_role<a name="gp_role"></a>
+
+The role of this server process " set to *dispatch* for the master and *execute* for a segment.
+
+<table>
+<colgroup>
+<col width="33%" />
+<col width="33%" />
+<col width="33%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Value Range</th>
+<th>Default</th>
+<th>Set Classifications</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>dispatch
+<p>execute</p>
+<p>utility</p></td>
+<td> </td>
+<td>read only</td>
+</tr>
+</tbody>
+</table>
+
+## gp\_safefswritesize<a name="gp_safefswritesize"></a>
+
+Specifies a minimum size for safe write operations to append-only tables in a non-mature file system. When a number of bytes greater than zero is specified, the append-only writer adds padding data up to that number in order to prevent data corruption due to file system errors. Each non-mature file system has a known safe write size that must be specified here when using HAWQ with that type of file system. This is commonly set to a multiple of the extent size of the file system; for example, Linux ext3 is 4096 bytes, so a value of 32768 is commonly used.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|------------------------|
+| integer | 0 | local, system, restart |
+
+## gp\_segment\_connect\_timeout<a name="gp_segment_connect_timeout"></a>
+
+Time that the HAWQ interconnect will try to connect to a segment instance over the network before timing out. Controls the network connection timeout between master and segment replication processes.
+
+| Value Range | Default | Set Classifications |
+|---------------------------------------------|---------|-----------------------|
+| Any valid time expression (number and unit) | 10min | local, system, reload |
+
+## gp\_segments\_for\_planner<a name="gp_segments_for_planner"></a>
+
+Sets the number of segment instances for the legacy query optimizer (planner) to assume in its cost and size estimates. If 0, then the value used is the actual number of segments. This variable affects the legacy optimizer's estimates of the number of rows handled by each sending and receiving process in Motion operators.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| integer | 0 | master, session, reload |
+
+## gp\_session\_id<a name="gp_session_id"></a>
+
+A system assigned ID number for a client session. Starts counting from 1 when the master instance is first started.
+
+| Value Range | Default | Set Classifications |
+|-----------------|---------|---------------------|
+| integer (1-*n*) | 14 | read only |
+
+## gp\_set\_proc\_affinity<a name="gp_set_proc_affinity"></a>
+
+If enabled, when a HAWQ server process (postmaster) is started it will bind to a CPU.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | off | master, system, restart |
+
+## gp\_set\_read\_only<a name="gp_set_read_only"></a>
+
+Set to on to disable writes to the database. Any in progress transactions must finish before read-only mode takes affect.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | off | master, session, reload |
+
+## gp\_statistics\_pullup\_from\_child\_partition<a name="gp_statistics_pullup_from_child_partition"></a>
+
+Enables the use of statistics from child tables when planning queries on the parent table by the legacy query optimizer (planner).
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, session, reload |
+
+## gp\_statistics\_use\_fkeys<a name="gp_statistics_use_fkeys"></a>
+
+When enabled, allows the legacy query optimizer (planner) to use foreign key information stored in the system catalog to optimize joins between foreign keys and primary keys.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, session, reload |
+
+## gp\_vmem\_protect\_segworker\_cache\_limit<a name="gp_vmem_protect_segworker_cache_limit"></a>
+
+If a query executor process consumes more than this configured amount, then the process will not be cached for use in subsequent queries after the process completes. Systems with lots of connections or idle processes may want to reduce this number to free more memory on the segments. Note that this is a local parameter and must be set for every segment.
+
+| Value Range | Default | Set Classifications |
+|---------------------|---------|------------------------|
+| number of megabytes | 500 | local, system, restart |
+
+## gp\_workfile\_checksumming<a name="gp_workfile_checksumming"></a>
+
+Adds a checksum value to each block of a work file (or spill file) used by `HashAgg` and `HashJoin` query operators. This adds an additional safeguard from faulty OS disk drivers writing corrupted blocks to disk. When a checksum operation fails, the query will cancel and rollback rather than potentially writing bad data to disk.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, session, reload |
+
+## gp\_workfile\_compress\_algorithm<a name="gp_workfile_compress_algorithm"></a>
+
+When a hash aggregation or hash join operation spills to disk during query processing, specifies the compression algorithm to use on the spill files. If using zlib, it must be in your $PATH on all segments.
+
+If your HAWQ database installation uses serial ATA (SATA) disk drives, setting the value of this parameter to `zlib` might help to avoid overloading the disk subsystem with IO operations.
+
+<table>
+<colgroup>
+<col width="33%" />
+<col width="33%" />
+<col width="33%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Value Range</th>
+<th>Default</th>
+<th>Set Classifications</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>none
+<p>zlib</p></td>
+<td>none</td>
+<td>master, session, reload</td>
+</tr>
+</tbody>
+</table>
+
+## gp\_workfile\_limit\_files\_per\_query<a name="gp_workfile_limit_files_per_query"></a>
+
+Sets the maximum number of temporary spill files (also known as workfiles) allowed per query per segment. Spill files are created when executing a query that requires more memory than it is allocated. The current query is terminated when the limit is exceeded.
+
+Set the value to 0 (zero) to allow an unlimited number of spill files. master session reload
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| integer | 3000000 | master, session, reload |
+
+## gp\_workfile\_limit\_per\_query<a name="gp_workfile_limit_per_query"></a>
+
+Sets the maximum disk size an individual query is allowed to use for creating temporary spill files at each segment. The default value is 0, which means a limit is not enforced.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| kilobytes | 0 | master, session, reload |
+
+## gp\_workfile\_limit\_per\_segment<a name="gp_workfile_limit_per_segment"></a>
+
+Sets the maximum total disk size that all running queries are allowed to use for creating temporary spill files at each segment. The default value is 0, which means a limit is not enforced.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|------------------------|
+| kilobytes | 0 | local, system, restart |
+
+## hawq\_dfs\_url<a name="hawq_dfs_url"></a>
+
+URL for HAWQ data directories on HDFS. The directory that you specify must be writeable by the `gpadmin` user. For example 'localhost:8020/hawq\_default'. If you have high availability enabled for your HDFS NameNodes, then this configuration parameter must be set to the service ID you configured in HDFS. See "HAWQ Filespaces and High Availability Enabled HDFS" for more information.
+
+| Value Range | Default | Set Classifications |
+|-------------------------------------------------------------------------|---------------------|-------------------------|
+| URL in the form of *NameNode\_host name*:*port*/*data\_directory\_name* | localhost:8020/hawq | master, session, reload |
+
+## hawq\_global\_rm\_type<a name="hawq_global_rm_type"></a>
+
+HAWQ global resource manager type. Valid values are `yarn` and `none`. Setting this parameter to `none` indicates that the HAWQ resource manager will manages its own resources. Setting the value to `yarn` means that HAWQ will negotiate with YARN for resources.
+
+| Value Range | Default | Set Classifications |
+|--------------|---------|-------------------------|
+| yarn or none | none | master, system, restart |
+
+## hawq\_master\_address\_host<a name="hawq_master_address_host"></a>
+
+Address or hostname of HAWQ master.
+
+| Value Range | Default | Set Classifications |
+|-----------------|-----------|-------------------------|
+| master hostname | localhost | master, session, reload |
+
+## hawq\_master\_address\_port<a name="hawq_master_address_port"></a>
+
+Port of the HAWQ master.
+
+| Value Range | Default | Set Classifications |
+|-------------------|---------|-------------------------|
+| valid port number | | master, session, reload |
+
+## hawq\_master\_directory<a name="hawq_master_directory"></a>
+
+Master server data directory.
+
+| Value Range | Default | Set Classifications |
+|----------------|---------|-------------------------|
+| directory name | | master, session, reload |
+
+## hawq\_master\_temp\_directory<a name="hawq_master_temp_directory"></a>
+
+One or more temporary directories for the HAWQ master. Separate multiple entries with commas.
+
+| Value Range | Default | Set Classifications |
+|-----------------------------------------------------------|---------|-------------------------|
+| directory name or comma-separated list of directory names | /tmp | master, session, reload |
+
+## hawq\_re\_memory\_overcommit\_max<a name="hawq_re_memory_overcommit_max"></a>
+
+Sets the maximum quota of memory overcommit (in MB) per physical segment for resource enforcement. This parameter sets the memory quota that can be overcommited beyond the memory quota dynamically assigned by the resource manager.
+
+Specify a larger value to prevent out of memory errors in YARN mode.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| integer | 8192 | master, system, restart |
+
+## hawq\_rm\_cluster\_report\_period<a name="hawq_rm_cluster_report"></a>
+
+Defines the time period, in seconds, for refreshing the global resource manager’s cluster report.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| 10-100 | 60 | master, session, reload |
+
+## hawq\_rm\_force\_alterqueue\_cancel\_queued\_request<a name="hawq_rm_force_alterqueue_cancel_queued_request"></a>
+
+Instructs HAWQ to cancel by all resource requests that are in conflict with the new resource queue definitions supplied in a ALTER RESOURCE QUEUE statement.
+
+If you set this parameter to false, the actions specified in ALTER RESOURCE QUEUE are canceled if the resource manager finds at least one resource request that is in conflict with the new resource definitions supplied in the altering command.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, session, reload |
+
+## hawq\_rm\_master\_port<a name="hawq_rm_master_port"></a>
+
+HAWQ resource manager master port number.
+
+| Value Range | Default | Set Classifications |
+|-------------------|---------|-------------------------|
+| valid port number | 5437 | master, session, reload |
+
+## hawq\_rm\_memory\_limit\_perseg<a name="hawq_rm_memory_limit_perseg"></a>
+
+Limit of memory usage by a HAWQ segment when `hawq_global_rm_type` is set to `none`. For example, `8GB`.
+
+| Value Range | Default | Set Classifications |
+|----------------------------------|---------|---------------------|
+| no specific lower or upper limit | 64GB | session, reload |
+
+## hawq\_rm\_min\_resource\_perseg<a name="hawq_rm_min_resource_perseg"></a>
+
+This property guarantees that the resource manager will acquire at the least the specified number of YARN containers for each segment no matter how small the resource request is for the initial query. This property cannot exceed YARN's queue capacity.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| 0 to 65535 | 2 | master, session, reload |
+
+## hawq\_rm\_nresqueue\_limit<a name="hawq_rm_nresqueue_limit"></a>
+
+Maximum number of resource queues that you can create. The minimum number you can create is 3 and the maximum is 1024.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| 3 to 1024 | 128 | master, session, reload |
+
+## hawq\_rm\_nslice\_perseg\_limit<a name="hawq_rm_nslice_perseg_limit"></a>
+
+Limits the total number of slices in one HAWQ segment. HAWQ QE execution generates multiple processes called slices to process data in parallel. Too many processes can affect OS performance. If you need to allow more processes in one segment to run in parallel for high concurrency due to high complexity, then you can modify this parameter.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| 1 to 65535 | 5000 | master, session, reload |
+
+## hawq\_rm\_nvcore\_limit\_perseg<a name="hawq_rm_nvcore_limit_perseg"></a>
+
+Maximum number of virtual cores that can be used for query execution in a HAWQ segment when `hawq_global_rm_type` is set to `none`. For example, `2.0`.
+
+| Value Range | Default | Set Classifications |
+|------------------------|---------|-------------------------|
+| 1.0 to maximum integer | 1.0 | master, session, reload |
+
+## hawq\_rm\_nvseg\_perquery\_limit<a name="hawq_rm_nvseg_perquery_limit"></a>
+
+Cluster-wide limit on the maximum number of virtual segments that can be used for one query statement execution. The hash buckets defined in `default_hash_table_bucket_number` cannot exceed this number.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| 1 to 65535 | 512 | master, session, reload |
+
+## hawq\_rm\_nvseg\_perquery\_perseg\_limit<a name="hawq_rm_nvseg_perquery_perseg_limit"></a>
+
+Cluster-wide limit on the maximum number of virtual segments in one HAWQ segment that can be used to execute one query statement. This parameter limits use of virtual segments for random tables, external tables, and user-defined tables, but has no effect on hash tables. If the query contains external tables or user-defined functions, it will allocate the number of segments defined by this parameter (value times segment count). Under most deployment circumstances, you should not modify the default.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| 1 to 65535 | 6 | master, session, reload |
+
+## hawq\_rm\_nvseg\_variance\_amon\_seg\_limit<a name="hawq_rm_nvseg_variance_amon_seg_limit"></a>
+
+Maximum variance allowed for the number of virtual segments per physical segment used to execute one query statement. Resource manager will try to reallocate resources to avoid going over this limit. This configuration is introduced to ensure that the virtual segments allocated for query execution are dispatched evenly to physical segments.
+
+For example, one query execution causes nine (9) virtual segments to be dispatched to two (2) physical segments. Assume that one segment has been allocated seven (7) virtual segments and another one has allocated two (2) virtual segments. Then the variance between the segments is five (5). If hawq\_rm\_nvseg\_variance\_amon\_seg\_limit is set to the default of one (1), then the allocation of resources for this query is rejected and will be reallocated later. However, if a physical segment has five virtual segments and the other physical segment has four, then this resource allocation is accepted.
+
+Queries perform best when virtual segments allocated for query execution are dispatched as evenly as possible across physical segments in the cluster. Therefore, if you are not concerned with a downgrade in query performance, you can increase the value of this configuration parameter to prevent the rejection of query resource allocations. Another case where you may want to modify this configuration is if you are using YARN to manage HAWQ resources and YARN cannot provide YARN containers to HAWQ evenly among hosts due to busy application workloads. In this case, the latency of resource allocations for queries can be large if this configuration parameter is too small.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|---------------------|
+| 0 to 65535 | 1 | master |
+
+## hawq\_rm\_rejectrequest\_nseg\_limit<a name="hawq_rm_rejectrequest_nseg_limit"></a>
+
+Maximum number of unavailable physical segments, expressed as a fraction of the entire cluster, that are allowed before the resource manager rejects query resource requests.
+
+Given an expected cluster size (as provided in the `slaves` file), the HAWQ resource manager calculates the maximum number of unavailable segments allowed before rejecting queries. For example, if `hawq_rm_rejectrequest_nseg_limit` is set to 0.25 and the expected cluster size is 15, then the maximum number of unavailable segments allowed is 4 (4=ceil(15 \* 0.25)). If the number of unavailable HAWQ physical segments exceeds the calculated number (for example, 5 in the preceding scenario) then the resource manager directly rejects query resource requests.
+
+In most deployment cases, you do not need to modify this value.
+
+**Note:** The value of [hawq\_rm\_rejectrequest\_nseg\_limit](#hawq_rm_rejectrequest_nseg_limit) should not exceed the value of [hawq\_rm\_tolerate\_nseg\_limit](#hawq_rm_tolerate_nseg_limit).
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|---------------------|
+| 0.0 to 1.0 | 0.25 | master |
+
+## hawq\_rm\_resource\_idle\_timeout<a name="hawq_rm_resource_idle_timeout"></a>
+
+Amount of time, in seconds, before idle resources are returned to YARN.
+
+| Value Range | Default | Set Classifications |
+|-----------------------|---------|-------------------------|
+| Seconds (-1 to 65535) | 300 | master, session, reload |
+
+## hawq\_rm\_return\_percent\_on\_overcommit<a name="hawq_rm_return_percent_on_overcommit"></a>
+
+Determines how many containers the global resource manager should return to the global resource manager (YARN for example.) This configuration only applies when HAWQ's YARN queue is busy, and HAWQ makes the YARN queue overuse its resources. The default value is 10, which means HAWQ will return 10% of acquired YARN containers by pausing the allocation of resources to HAWQ queries.
+
+In a typical deployment, you do not need to modify the default value of this parameter.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| 0-100 | 10 | master, session, reload |
+
+## hawq\_rm\_segment\_heartbeat\_interval<a name="hawq_rm_segment_heartbeat_interval"></a>
+
+Time interval, in seconds, that the segment uses to send a heartbeat to the resource manager. This heartbeat keeps the segment alive and presents the segment's latest status.
+
+For most deployments, you do not need to modify the default value.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|---------------------|
+| 1 to 65535 | 30 | master |
+
+## hawq\_rm\_segment\_port<a name="hawq_rm_segment_port"></a>
+
+HAWQ resource manager segment server port number.
+
+| Value Range | Default | Set Classifications |
+|-------------------|---------|-------------------------|
+| valid port number | 5438 | master, session, reload |
+
+## hawq\_rm\_stmt\_nvseg<a name="hawq_rm_stmt_nvseg"></a>
+
+Defines the number of virtual segments to use for the next query's execution. The default value is 0 which means statement level resource quota setting is disabled.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| 0 to 65535 | 0 | master, session, reload |
+
+## hawq\_rm\_stmt\_vseg\_memory<a name="hawq_rm_stmt_vseg_memory"></a>
+
+Defines the memory quota of one virtual segment.
+
+| Value Range | Default | Set Classifications |
+|-------------------------------------------------------------------------------------|---------|-------------------------|
+| Acceptable values are ‘128mb’, ’256mb’, ’512mb’, ‘1gb’, ‘2gb’, ‘4gb’, ‘8gb’, ‘16gb' | 128mb | master, session, reload |
+
+## hawq\_rm\_tolerate\_nseg\_limit<a name="hawq_rm_tolerate_nseg_limit"></a>
+
+Maximum number of unused physical segments allowed when executing a query statement, expressed as a fraction of the total number of segments cluster. HAWQ checks this limit if a query has at least one physical segment with more than one virtual segments allocated. This configuration ensures that when executing a query in HAWQ, all allocated virtual segments are dispatched as widely as possible across physical segments.
+
+Given an expected cluster size (as indicated in the `slaves` file), the HAWQ resource manager calculates the tolerated number of unused segments. For example, if `hawq_rm_tolerate_nseg_limit` is set to 0.25 and the expected cluster size is 15, the tolerated number of unused segments for a query is 4 (4=ceil(15 \* 0.25)). If the number of unused segments exceeds the calculated value, the resource manager temporarily gives up on allocating resources for the query and will keep retrying until this limit is no longer exceeded.
+
+Under most deployment circumstances, you do not need to modify the default. Reasons why you would want to increase this number include:
+
+- You have an unstable network environment and expect to have a large quantity of unavailable machines. Increase this configuration value to avoid large query resource allocation latency.
+
+**Note:** The value of [hawq\_rm\_rejectrequest\_nseg\_limit](#hawq_rm_rejectrequest_nseg_limit) should not exceed the value of [hawq\_rm\_tolerate\_nseg\_limit](#hawq_rm_tolerate_nseg_limit).
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|---------------------|
+| 0.0 to 1.0 | 0.25 | master |
+
+## hawq\_rm\_yarn\_address<a name="hawq_rm_yarn_address"></a>
+
+Server address (host and port) of the YARN resource manager server (the value of `yarn.resourcemanager.address`). User must define this if `hawq_global_rm_type` is set to `yarn`. For example, `localhost:8032`.
+
+**Note:** If you have enabled high availability for your YARN resource managers, then you must configure `yarn.resourcemanager.ha` and `yarn.resourcemanager.scheduler.ha` in `yarn-client.xml` located in `$GPHOME/etc` instead. The values specified for `hawq_rm_yarn_address` and `hawq_rm_yarn_scheduler_address` are ignored.
+
+| Value Range | Default | Set Classifications |
+|-------------------------|---------|---------------------|
+| valid hostname and port | | master |
+
+## hawq\_rm\_yarn\_app\_name<a name="hawq_rm_yarn_app_name"></a>
+
+The name of the YARN application registered with HAWQ's resource manager. For example, `hawq`.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|---------------------|
+| string | hawq | master |
+
+## hawq\_rm\_yarn\_queue\_name<a name="hawq_rm_yarn_queue_name"></a>
+
+The name of the YARN resource queue to register with HAWQ's resource manager.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|---------------------|
+| string | default | master |
+
+## hawq\_rm\_yarn\_scheduler\_address<a name="hawq_rm_yarn_scheduler_address"></a>
+
+Server address (host and port) of the YARN resource manager scheduler (the value of `yarn.resourcemanager.scheduler.address`). User must define this if `hawq_global_rm_type` is set to `yarn`. For example, `localhost:8030`.
+
+**Note:** If you have enabled high availability for your YARN resource managers, then you must configure `yarn.resourcemanager.ha` and `yarn.resourcemanager.scheduler.ha` in `yarn-client.xml` located in `$GPHOME/etc` instead. The values specified for `hawq_rm_yarn_address` and `hawq_rm_yarn_scheduler_address` are ignored.
+
+| Value Range | Default | Set Classifications |
+|-------------------------|---------|---------------------|
+| valid hostname and port | | master |
+
+## hawq\_segment\_address\_port<a name="hawq_segment_address_port"></a>
+
+Base port for the HAWQ segment host.
+
+| Value Range | Default | Set Classifications |
+|-------------------|---------|------------------------|
+| valid port number | | local, session, reload |
+
+## hawq\_segment\_directory<a name="hawq_segment_directory"></a>
+
+Base path for the HAWQ segment data directory.
+
+| Value Range | Default | Set Classifications |
+|----------------|---------|------------------------|
+| directory name | | local, session, reload |
+
+## hawq\_segment\_temp\_directory<a name="hawq_segment_temp_directory"></a>
+
+One or more temporary directories for the HAWQ segment. Separate multiple entries with commas.
+
+| Value Range | Default | Set Classifications |
+|-----------------------------------------------------------|---------|------------------------|
+| directory name or comma-separated list of directory names | /tmp | local, session, reload |
+
+## integer\_datetimes<a name="integer_datetimes"></a>
+
+Reports whether PostgreSQL was built with support for 64-bit-integer dates and times.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|---------------------|
+| Boolean | on | read only |
+
+## IntervalStyle<a name="IntervalStyle"></a>
+
+Sets the display format for interval values. The value *sql\_standard* produces output matching SQL standard interval literals. The value *postgres* produces output matching PostgreSQL releases prior to 8.4 when the [DateStyle](#DateStyle) parameter was set to ISO.
+
+The value *iso\_8601* will produce output matching the time interval *format with designators* defined in section 4.4.3.2 of ISO 8601. See the [<span class="ph">PostgreSQL 8.4 documentation</span>](http://www.postgresql.org/docs/8.4/static/datatype-datetime.html) for more information.
+
+<table>
+<colgroup>
+<col width="33%" />
+<col width="33%" />
+<col width="33%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Value Range</th>
+<th>Default</th>
+<th>Set Classifications</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>postgres
+<p>postgres_verbose</p>
+<p>sql_standard</p>
+<p>iso_8601</p></td>
+<td>postgres</td>
+<td>master, session, reload</td>
+</tr>
+</tbody>
+</table>
+
+## join\_collapse\_limit<a name="join_collapse_limit"></a>
+
+The legacy query optimizer (planner) will rewrite explicit inner `JOIN` constructs into lists of `FROM` items whenever a list of no more than this many items in total would result. By default, this variable is set the same as *from\_collapse\_limit*, which is appropriate for most uses. Setting it to 1 prevents any reordering of inner JOINs. Setting this variable to a value between 1 and *from\_collapse\_limit* might be useful to trade off planning time against the quality of the chosen plan (higher values produce better plans).
+
+| Value Range | Default | Set Classifications |
+|-----------------|---------|-------------------------|
+| integer (1-*n*) | 20 | master, session, reload |
+
+## krb\_caseins\_users<a name="krb_caseins_users"></a>
+
+Sets whether Kerberos user names should be treated case-insensitively. The default is case sensitive (off).
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | off | master, system, restart |
+
+## krb\_server\_keyfile<a name="krb_server_keyfile"></a>
+
+Sets the location of the Kerberos server key file.
+
+| Value Range | Default | Set Classifications |
+|--------------------|---------|-------------------------|
+| path and file name | unset | master, system, restart |
+
+## krb\_srvname<a name="krb_srvname"></a>
+
+Sets the Kerberos service name.
+
+| Value Range | Default | Set Classifications |
+|--------------|----------|-------------------------|
+| service name | postgres | master, system, restart |
+
+## lc\_collate<a name="lc_collate"></a>
+
+Reports the locale in which sorting of textual data is done. The value is determined when the HAWQ array is initialized.
+
+| Value Range | Default | Set Classifications |
+|--------------------------|---------|---------------------|
+| <system dependent> | | read only |
+
+## lc\_ctype<a name="lc_ctype"></a>
+
+Reports the locale that determines character classifications. The value is determined when the HAWQ array is initialized.
+
+| Value Range | Default | Set Classifications |
+|--------------------------|---------|---------------------|
+| <system dependent> | | read only |
+
+## lc\_messages<a name="lc_messages"></a>
+
+Sets the language in which messages are displayed. The locales available depends on what was installed with your operating system - use *locale -a* to list available locales. The default value is inherited from the execution environment of the server. On some systems, this locale category does not exist. Setting this variable will still work, but there will be no effect. Also, there is a chance that no translated messages for the desired language exist. In that case you will continue to see the English messages.
+
+| Value Range | Default | Set Classifications |
+|--------------------------|---------|------------------------|
+| <system dependent> | | local, system, restart |
+
+## lc\_monetary<a name="lc_monetary"></a>
+
+Sets the locale to use for formatting monetary amounts, for example with the *to\_char* family of functions. The locales available depends on what was installed with your operating system - use *locale -a* to list available locales. The default value is inherited from the execution environment of the server.
+
+| Value Range | Default | Set Classifications |
+|--------------------------|---------|------------------------|
+| <system dependent> | | local, system, restart |
+
+## lc\_numeric<a name="lc_numeric"></a>
+
+Sets the locale to use for formatting numbers, for example with the *to\_char* family of functions. The locales available depends on what was installed with your operating system - use *locale -a* to list available locales. The default value is inherited from the execution environment of the server.
+
+| Value Range | Default | Set Classifications |
+|--------------------------|---------|------------------------|
+| <system dependent> | | local, system, restart |
+
+## lc\_time<a name="lc_time"></a>
+
+This parameter currently does nothing, but may in the future.
+
+| Value Range | Default | Set Classifications |
+|--------------------------|---------|------------------------|
+| <system dependent> | | local, system, restart |
+
+## listen\_addresses<a name="listen_addresses"></a>
+
+Specifies the TCP/IP address(es) on which the server is to listen for connections from client applications - a comma-separated list of host names and/or numeric IP addresses. The special entry \* corresponds to all available IP interfaces. If the list is empty, only UNIX-domain sockets can connect.
+
+<table>
+<colgroup>
+<col width="33%" />
+<col width="33%" />
+<col width="33%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Value Range</th>
+<th>Default</th>
+<th>Set Classifications</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>localhost,
+<p>host names,</p>
+<p>IP addresses,</p>
+<p>* (all available IP interfaces)</p></td>
+<td>*</td>
+<td>master, system, restart</td>
+</tr>
+</tbody>
+</table>
+
+## local\_preload\_libraries<a name="local_preload_libraries"></a>
+
+Comma separated list of shared library files to preload at the start of a client session.
+
+| Value Range | Default | Set Classifications |
+|-------------------------------|--------------|------------------------|
+| string (comma-separated list) | no value set | local, system, restart |
+
+## log\_autostats<a name="log_autostats"></a>
+
+Logs information about automatic `ANALYZE` operations related to [gp\_autostats\_mode](#gp_autostats_mode).
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|------------------------------------|
+| Boolean | off | master, session, reload, superuser |
+
+## log\_connections<a name="log_connections"></a>
+
+This outputs a line to the server log detailing each successful connection. Some client programs, like psql, attempt to connect twice while determining if a password is required, so duplicate "connection received" messages do not always indicate a problem.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|------------------------|
+| Boolean | off | local, system, restart |
+
+## log\_disconnections<a name="log_disconnections"></a>
+
+This outputs a line in the server log at termination of a client session, and includes the duration of the session.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|------------------------|
+| Boolean | off | local, system, restart |
+
+## log\_dispatch\_stats<a name="log_dispatch_stats"></a>
+
+When set to "on," this parameter adds a log message with verbose information about the dispatch of the statement.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|------------------------|
+| Boolean | off | local, system, restart |
+
+## log\_duration<a name="log_duration"></a>
+
+Causes the duration of every completed statement which satisfies *log\_statement* to be logged.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|------------------------------------|
+| Boolean | off | master, session, reload, superuser |
+
+## log\_error\_verbosity<a name="log_error_verbosity"></a>
+
+Controls the amount of detail written in the server log for each message that is logged.
+
+<table>
+<colgroup>
+<col width="33%" />
+<col width="33%" />
+<col width="33%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Value Range</th>
+<th>Default</th>
+<th>Set Classifications</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>TERSE
+<p>DEFAULT</p>
+<p>VERBOSE</p></td>
+<td>DEFAULT</td>
+<td>master, session, reload, superuser</td>
+</tr>
+</tbody>
+</table>
+
+## log\_executor\_stats<a name="log_executor_stats"></a>
+
+For each query, write performance statistics of the query executor to the server log. This is a crude profiling instrument. Cannot be enabled together with *log\_statement\_stats*.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|------------------------|
+| Boolean | off | local, system, restart |
+
+## log\_hostname<a name="log_hostname"></a>
+
+By default, connection log messages only show the IP address of the connecting host. Turning on this option causes logging of the IP address and host name of the HAWQ master. Note that depending on your host name resolution setup this might impose a non-negligible performance penalty.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | off | master, system, restart |
+
+## log\_min\_duration\_statement<a name="log_min_duration_statement"></a>
+
+Logs the statement and its duration on a single log line if its duration is greater than or equal to the specified number of milliseconds. Setting this to 0 will print all statements and their durations. -1 disables the feature. For example, if you set it to 250 then all SQL statements that run 250ms or longer will be logged. Enabling this option can be useful in tracking down unoptimized queries in your applications.
+
+| Value Range | Default | Set Classifications |
+|-------------------------------|---------|------------------------------------|
+| number of milliseconds, 0, -1 | -1 | master, session, reload, superuser |
+
+## log\_min\_error\_statement<a name="log_min_error_statement"></a>
+
+Controls whether or not the SQL statement that causes an error condition will also be recorded in the server log. All SQL statements that cause an error of the specified level or higher are logged. The default is PANIC (effectively turning this feature off for normal use). Enabling this option can be helpful in tracking down the source of any errors that appear in the server log.
+
+<table>
+<colgroup>
+<col width="33%" />
+<col width="33%" />
+<col width="33%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Value Range</th>
+<th>Default</th>
+<th>Set Classifications</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>DEBUG5
+<p>DEBUG4</p>
+<p>DEBUG3</p>
+<p>DEBUG2</p>
+<p>DEBUG1</p>
+<p>INFO</p>
+<p>NOTICE</p>
+<p>WARNING</p>
+<p>ERROR</p>
+<p>FATAL</p>
+<p>PANIC</p></td>
+<td>ERROR</td>
+<td>master, session, reload, superuser</td>
+</tr>
+</tbody>
+</table>
+
+## log\_min\_messages<a name="log_min_messages"></a>
+
+Controls which message levels are written to the server log. Each level includes all the levels that follow it. The later the level, the fewer messages are sent to the log.
+
+<table>
+<colgroup>
+<col width="33%" />
+<col width="33%" />
+<col width="33%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Value Range</th>
+<th>Default</th>
+<th>Set Classifications</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>DEBUG5
+<p>DEBUG4</p>
+<p>DEBUG3</p>
+<p>DEBUG2</p>
+<p>DEBUG1</p>
+<p>INFO</p>
+<p>NOTICE</p>
+<p>WARNING</p>
+<p>ERROR</p>
+<p>FATAL</p>
+<p>PANIC</p></td>
+<td>WARNING</td>
+<td>master, session, reload, superuser</td>
+</tr>
+</tbody>
+</table>
+
+## log\_parser\_stats<a name="log_parser_stats"></a>
+
+For each query, write performance statistics of the query parser to the server log. This is a crude profiling instrument. Cannot be enabled together with *log\_statement\_stats*.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|------------------------------------|
+| Boolean | off | master, session, reload, superuser |
+
+## log\_planner\_stats<a name="log_planner_stats"></a>
+
+For each query, write performance statistics of the legacy query optimizer (planner) to the server log. This is a crude profiling instrument. Cannot be enabled together with *log\_statement\_stats*.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|------------------------------------|
+| Boolean | off | master, session, reload, superuser |
+
+## log\_rotation\_age<a name="log_rotation_age"></a>
+
+Determines the maximum lifetime of an individual log file. After this time has elapsed, a new log file will be created. Set to zero to disable time-based creation of new log files.
+
+| Value Range | Default | Set Classifications |
+|---------------------------------------------|---------|------------------------|
+| Any valid time expression (number and unit) | 1d | local, system, restart |
+
+## log\_rotation\_size<a name="log_rotation_size"></a>
+
+Determines the maximum size of an individual log file. After this many kilobytes have been emitted into a log file, a new log file will be created. Set to zero to disable size-based creation of new log files.
+
+The maximum value is INT\_MAX/1024. If an invalid value is specified, the default value is used. INT\_MAX is the largest value that can be stored as an integer on your system.
+
+| Value Range | Default | Set Classifications |
+|---------------------|---------|------------------------|
+| number of kilobytes | 0 | local, system, restart |
+
+## log\_statement<a name="log_statement"></a>
+
+Controls which SQL statements are logged. DDL logs all data definition commands like CREATE, ALTER, and DROP commands. MOD logs all DDL statements, plus INSERT, UPDATE, DELETE, TRUNCATE, and COPY FROM. PREPARE and EXPLAIN ANALYZE statements are also logged if their contained command is of an appropriate type.
+
+<table>
+<colgroup>
+<col width="33%" />
+<col width="33%" />
+<col width="33%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Value Range</th>
+<th>Default</th>
+<th>Set Classifications</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>NONE
+<p>DDL</p>
+<p>MOD</p>
+<p>ALL</p></td>
+<td>ALL</td>
+<td>master, session, reload, superuser</td>
+</tr>
+</tbody>
+</table>
+
+## log\_statement\_stats<a name="log_statement_stats"></a>
+
+For each query, write total performance statistics of the query parser, planner, and executor to the server log. This is a crude profiling instrument.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|------------------------------------|
+| Boolean | off | master, session, reload, superuser |
+
+## log\_timezone<a name="log_timezone"></a>
+
+Sets the time zone used for timestamps written in the log. Unlike [TimeZone](#TimeZone), this value is system-wide, so that all sessions will report timestamps consistently. The default is `unknown`, which means to use whatever the system environment specifies as the time zone.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|------------------------|
+| string | unknown | local, system, restart |
+
+## log\_truncate\_on\_rotation<a name="log_truncate_on_rotation"></a>
+
+Truncates (overwrites), rather than appends to, any existing log file of the same name. Truncation will occur only when a new file is being opened due to time-based rotation. For example, using this setting in combination with a log\_filename such as `hawq-%H.log` would result in generating twenty-four hourly log files and then cyclically overwriting them. When off, pre-existing files will be appended to in all cases.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|------------------------|
+| Boolean | off | local, system, restart |
+
+## max\_appendonly\_tables<a name="max_appendonly_tables"></a>
+
+Sets the maximum number of append-only relations that can be written to or loaded concurrently. Append-only table partitions and subpartitions are considered as unique tables against this limit. Increasing the limit will allocate more shared memory at server start.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| integer | 10000 | master, system, restart |
+
+## max\_connections<a name="max_connections"></a>
+
+The maximum number of concurrent connections allowed on master. In a HAWQ system, user client connections go through the HAWQ master instance only. Segment instances should allow 5-10 times the amount as the master. When you increase this parameter, you must increase [max\_prepared\_transactions](#max_prepared_transactions) as well. For more information about limiting concurrent connections, see [Configuring Client Authentication](/20/clientaccess/client_auth.html).
+
+Increasing this parameter may cause HAWQ to request more shared memory. See [shared\_buffers](#shared_buffers) for information about HAWQ server instance shared memory buffers.
+
+| Value Range | Default | Set Classifications |
+|------------------|---------|------------------------|
+| integer (10-*n*) | 1280 | local, system, restart |
+
+## max\_files\_per\_process<a name="max_files_per_process"></a>
+
+Sets the maximum number of simultaneously open files allowed to each server subprocess. If the kernel is enforcing a safe per-process limit, you don't need to worry about this setting. Some platforms such as BSD, the kernel will allow individual processes to open many more files than the system can really support.
+
+| Value Range | Default | Set Classifications |
+|------------------|---------|------------------------|
+| integer (40-*n*) | 150 | local, system, restart |
+
+## max\_fsm\_pages<a name="max_fsm_pages"></a>
+
+Sets the maximum number of disk pages for which free space will be tracked in the shared free-space map. Six bytes of shared memory are consumed for each page slot.
+
+| Value Range | Default | Set Classifications |
+|------------------------------------------|---------|------------------------|
+| integer > 16 \* *max\_fsm\_relations* | 200000 | local, system, restart |
+
+## max\_fsm\_relations<a name="max_fsm_relations"></a>
+
+Sets the maximum number of relations for which free space will be tracked in the shared memory free-space map. Should be set to a value larger than the total number of:
+
+tables + indexes + system tables.
+
+It costs about 60 bytes of memory for each relation per segment instance. It is better to allow some room for overhead and set too high rather than too low.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|------------------------|
+| integer | 1000 | local, system, restart |
+
+## max\_function\_args<a name="max_function_args"></a>
+
+Reports the maximum number of function arguments.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|---------------------|
+| integer | 100 | read only |
+
+## max\_identifier\_length<a name="max_identifier_length"></a>
+
+Reports the maximum identifier length.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|---------------------|
+| integer | 63 | read only |
+
+## max\_index\_keys<a name="max_index_keys"></a>
+
+Reports the maximum number of index keys.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|---------------------|
+| integer | 32 | read only |
+
+## max\_locks\_per\_transaction<a name="max_locks_per_transaction"></a>
+
+The shared lock table is created with room to describe locks on *max\_locks\_per\_transaction* \* (*max\_connections* + *max\_prepared\_transactions*) objects, so no more than this many distinct objects can be locked at any one time. This is not a hard limit on the number of locks taken by any one transaction, but rather a maximum average value. You might need to raise this value if you have clients that touch many different tables in a single transaction.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|------------------------|
+| integer | 128 | local, system, restart |
+
+## max\_prepared\_transactions<a name="max_prepared_transactions"></a>
+
+Sets the maximum number of transactions that can be in the prepared state simultaneously. HAWQ uses prepared transactions internally to ensure data integrity across the segments. This value must be at least as large as the value of [max\_connections](#max_connections) on the master. Segment instances should be set to the same value as the master.
+
+<table>
+<colgroup>
+<col width="33%" />
+<col width="33%" />
+<col width="33%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Value Range</th>
+<th>Default</th>
+<th>Set Classifications</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>integer</td>
+<td>250 on master
+<p>250 on segments</p></td>
+<td>local, system, restart</td>
+</tr>
+</tbody>
+</table>
+
+## max\_stack\_depth<a name="max_stack_depth"></a>
+
+Specifies the maximum safe depth of the server's execution stack. The ideal setting for this parameter is the actual stack size limit enforced by the kernel (as set by *ulimit -s* or local equivalent), less a safety margin of a megabyte or so. Setting the parameter higher than the actual kernel limit will mean that a runaway recursive function can crash an individual backend process.
+
+| Value Range | Default | Set Classifications |
+|---------------------|---------|------------------------|
+| number of kilobytes | 2MB | local, system, restart |
+
+## optimizer<a name="optimizer"></a>
+
+Enables the GPORCA when running SQL queries. The default is `on`. When `off`, HAWQ uses only the legacy query optimizer.
+
+GPORCA co-exists with the legacy query optimizer. When GPORCA is enabled, HAWQ uses GPORCA to generate an execution plan for a query when possible. If GPORCA cannot be used, the legacy query optimizer is used.
+
+The `optimizer` parameter can be set for a database system, an individual database, or a session or query.
+
+For information about the legacy query optimizer and GPORCA, see [About GPORCA](../../query/gporca/query-gporca-optimizer.html#topic_i4y_prl_vp).
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, session, reload |
+
+## optimizer\_analyze\_root\_partition<a name="optimizer_analyze_root_partition"></a>
+
+For a partitioned table, collects statistics for the root partition when the `ANALYZE` command is run on the table. GPORCA uses the root partition statistics when generating a query plan. The legacy query optimizer does not use these statistics. If you set the value of the server configuration parameter [optimizer](#optimizer) to on, set the value of this parameter to on and run the command `ANALYZE` or `ANALYZE ROOTPARTITION` on partitioned tables to ensure the proper statistics have been collected.
+
+For information about the legacy query optimizer and GPORCA, see [About GPORCA](../../query/gporca/query-gporca-optimizer.html#topic_i4y_prl_vp).
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, session, reload |
+
+## optimizer\_minidump<a name="optimizer_minidump"></a>
+
+GPORCA generates minidump files to describe the optimization context for a given query. Use the minidump files to analyze HAWQ issues. The minidump file is located under the master data directory and uses the following naming format:
+
+`Minidump_date_time.mdp`
+
+The minidump file contains this query related information:
+
+- Catalog objects including data types, tables, operators, and statistics required by GPORCA
+- An internal representation (DXL) of the query
+- An internal representation (DXL) of the plan produced by GPORCA
+- System configuration information passed to GPORCA such as server configuration parameters, cost and statistics configuration, and number of segments
+- A stack trace of errors generated while optimizing the query
+
+Setting this parameter to `ALWAYS` generates a minidump for all queries.
+
+**Note:** Set this parameter to `ONERROR` in production environments to minimize total optimization time.
+
+<table>
+<colgroup>
+<col width="33%" />
+<col width="33%" />
+<col width="33%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Value Range</th>
+<th>Default</th>
+<th>Set Classifications</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>ONERROR
+<p>ALWAYS</p></td>
+<td>ONERROR</td>
+<td>master, session, reload</td>
+</tr>
+</tbody>
+</table>
+
+## optimizer\_parts\_to\_force\_sort\_on\_insert<a name="optimizer_parts_to_force_sort_on_insert"></a>
+
+Sets the minimum number of partitions required to force GPORCA to generate a plan for sorting tuples during insertion into an append-only, row-oriented (AORO) partitioned tables. If the number of partitions is less than the specified value, no sorting is performed.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| integer | 160 | master, session, reload |
+
+## optimizer\_prefer\_scalar\_dqa\_multistage\_agg<a name="optimizer_prefer_scalar_dqa_multistage_agg"></a>
+
+When enabled, this parameter forces the GPORCA to use a three-stage scalar DQA plan. This type of plan evaluates the aggregate functions in three stages (local, intermediate, and global aggregations). By forcing the use of this plan, GPORCA ensures predictable performance for queries that contain distinct qualified aggregates (DQA).
+
+By default, this configuration is on. If you set this parameter to false, then GPORCA makes a cost-based decision.
+
+| Value Range | Default | Set Classifications |
+|-------------|-----------|-------------------------|
+| Boolean | true (on) | master, session, reload |
+
+## password\_encryption<a name="password_encryption"></a>
+
+When a password is specified in `CREATE USER` or `ALTER USER` without writing either `ENCRYPTED` or `UNENCRYPTED`, this option determines whether the password is to be encrypted.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, session, reload |
+
+## pgstat\_track\_activity\_query\_size<a name="pgstat_track_activity_query_size"></a>
+
+Sets the maximum length limit for the query text stored in `current_query` column of the system catalog view pg\_stat\_activity. The minimum length is 1024 characters.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|------------------------|
+| integer | 1024 | local, system, restart |
+
+## pljava\_classpath<a name="pljava_classpath"></a>
+
+A colon (:) separated list of the jar files containing the Java classes used in any PL/Java functions. The jar files listed here must also be installed on all HAWQ hosts in the following location: `$GPHOME/lib/postgresql/java/`
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| string | | master, session, reload |
+
+## pljava\_statement\_cache\_size<a name="pljava_statement_cache_size"></a>
+
+Sets the size in KB of the JRE MRU (Most Recently Used) cache for prepared statements.
+
+| Value Range | Default | Set Classifications |
+|---------------------|---------|------------------------------------|
+| number of kilobytes | 10 | master, system, restart, superuser |
+
+## pljava\_release\_lingering\_savepoints<a name="pljava_release_lingering_savepoints"></a>
+
+If true, lingering savepoints used in PL/Java functions will be released on function exit. If false, savepoints will be rolled back.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|------------------------------------|
+| Boolean | true | master, system, restart, superuser |
+
+## pljava\_vmoptions<a name="pljava_vmoptions"></a>
+
+Defines the startup options for the Java VM. The default value is an empty string ("").
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|------------------------------------|
+| string | | master, system, restart, superuser |
+
+## port<a name="port"></a>
+
+The database listener port for a HAWQ instance. The master and each segment has its own port. Port numbers for the HAWQ system must also be changed in the `gp_segment_configuration` catalog. You must shut down your HAWQ system before changing port numbers.
+
+| Value Range | Default | Set Classifications |
+|-----------------------|---------|------------------------|
+| any valid port number | 5432 | local, system, restart |
+
+## pxf\_enable\_stat\_collection<a name="pxf_enable_stat_collection"></a>
+
+Determines whether `ANALYZE` collects statistics for readable PXF tables. If true, an analyze operation on a readable PXF table estimates the number of tuples in the table, generates a sample table and runs statistics queries on the sample table to calculate statistics, as it does for native tables. When false, `ANALYZE` displays a message to warn that it is skipping the table because `pxf_enable_stat_collection` is off.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | true | master, session, reload |
+
+## pxf\_service\_address<a name="pxf_service_address"></a>
+
+Specifies the location and port of the PXF service used by HCatalog. To learn more about HCatalog querying by PXF, see [Using PXF and HCatalog to Query Hive](../../pxf/HivePXF.html#hcatalog).
+
+| Value Range | Default | Set Classifications |
+|---------------------------------------------------------|-----------------|-------------------------|
+| any valid hostname or IP address: any valid port number | localhost:51200 | master, session, reload |
+
+## pxf\_service\_port<a name="pxf_service_port"></a>
+
+Specifies the listen port for the PXF service when HDFS NameNode High Availability (HA) is enabled. PXF connects to this port when the port number is omitted from the `LOCATION` clause of a `CREATE EXTERNAL TABLE` command.
+
+| Value Range | Default | Set Classifications |
+|-----------------------|---------|-------------------------|
+| any valid port number | 51200 | master, session, reload |
+
+## pxf\_stat\_max\_fragments<a name="pxf_stat_max_fragments"></a>
+
+The maximum number of fragments to sample during `ANALYZE` on a PXF table. A lower value provides better performance for the `ANALYZE` command, but the sampling is less uniform. A higher value provides more uniform sampling, with slower `ANALYZE` performance.
+
+| Value Range | Default | Set Classifications |
+|---------------------|---------|-------------------------|
+| number of fragments | 100 | master, session, reload |
+
+## random\_page\_cost<a name="random_page_cost"></a>
+
+Sets the estimate of the cost of a nonsequentially fetched disk page for the legacy query optimizer (planner). This is measured as a multiple of the cost of a sequential page fetch. A higher value makes it more likely a sequential scan will be used, a lower value makes it more likely an index scan will be used.
+
+| Value Range | Default | Set Classifications |
+|----------------|---------|-------------------------|
+| floating point | 100 | master, session, reload |
+
+## regex\_flavor<a name="regex_flavor"></a>
+
+The 'extended' setting may be useful for exact backwards compatibility with pre-7.4 releases of PostgreSQL.
+
+<table>
+<colgroup>
+<col width="33%" />
+<col width="33%" />
+<col width="33%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Value Range</th>
+<th>Default</th>
+<th>Set Classifications</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>advanced
+<p>extended</p>
+<p>basic</p></td>
+<td>advanced</td>
+<td>master, session, reload</td>
+</tr>
+</tbody>
+</table>
+
+## runaway\_detector\_activation\_percent<a name="runaway_detector_activation_percent"></a>
+
+Sets the percentage of the virtual memory quota that will trigger the termination of queries. If the amount of virtual memory utilized by a physical segment exceeds the calculated threshold, then HAWQ begins terminating queries based on memory usage, starting with the query that is consuming the largest amount of memory. Queries are terminated until the percentage of utilized virtual memory is below the specified percentage.
+
+The amount of virtual memory available to a physical segment is calculated dynamically in HAWQ. HAWQ's resource manager determines resource allocations and quotas based on resources available to the cluster, resource queue configuration and other deployment configuration factors.
+
+To calculate the memory usage threshold for runaway queries, HAWQ uses the following formula:
+
+*vmem threshold* = (*virtual memory quota calculated by resource manager* + [hawq\_re\_memory\_overcommit\_max](#hawq_re_memory_overcommit_max)) \* **runaway\_detector\_activation\_percent**.
+
+For example, if HAWQ resource manager calculates a virtual memory quota of 9GB,` hawq_re_memory_overcommit_max` is set to 1GB and the value of `runaway_detector_activation_percent` is 95 (95%), then HAWQ starts terminating queries when the utilized virtual memory exceeds 9.5 GB.
+
+A value of 100 disables the automatic detection and termination of runaway queries.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| 0 to 100 | 95 | master, session, reload |
+
+## search\_path<a name="search_path"></a>
+
+Specifies the order in which schemas are searched when an object is referenced by a simple name with no schema component. When there are objects of identical names in different schemas, the one found first in the search path is used. The system catalog schema, *pg\_catalog*, is always searched, whether it is mentioned in the path or not. When objects are created without specifying a particular target schema, they will be placed in the first schema listed in the search path. The current effective value of the search path can be examined via the SQL function *current\_schemas()*. *current\_schemas()* shows how the requests appearing in *search\_path* were resolved.
+
+| Value Range | Default | Set Classifications |
+|----------------------------------------|--------------|-------------------------|
+| a comma-separated list of schema names | $user,public | master, session, reload |
+
+## seg\_max\_connections<a name="seg_max_connections"></a>
+
+The maximum number of concurrent connections on a segment. In a HAWQ system, user client connections go through the HAWQ master instance only. Segment instances should allow 5-10 times the amount of connections allowed on the master (see [max\_connections](#max_connections).) When you increase this parameter, you must increase [max\_prepared\_transactions](#max_prepared_transactions) as well. For more information about limiting concurrent connections, see [Configuring Client Authentication](/20/clientaccess/client_auth.html).
+
+Increasing this parameter may cause HAWQ to request more shared memory. See [shared\_buffers](#shared_buffers) for information about HAWQ server instance shared memory buffers.
+
+<table>
+<colgroup>
+<col width="33%" />
+<col width="33%" />
+<col width="33%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Value Range</th>
+<th>Default</th>
+<th>Set Classifications</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>integer (240-<em>n</em>)</td>
+<td><p>1280</p></td>
+<td>local, system, restart</td>
+</tr>
+</tbody>
+</table>
+
+## seq\_page\_cost<a name="seq_page_cost"></a>
+
+For the legacy query optimizer (planner), sets the estimate of the cost of a disk page fetch that is part of a series of sequential fetches.
+
+| Value Range | Default | Set Classifications |
+|----------------|---------|-------------------------|
+| floating point | 1 | master, session, reload |
+
+## server\_encoding<a name="server_encoding"></a>
+
+Reports the database encoding (character set). It is determined when the HAWQ array is initialized. Ordinarily, clients need only be concerned with the value of *client\_encoding*.
+
+| Value Range | Default | Set Classifications |
+|--------------------------|---------|---------------------|
+| <system dependent> | UTF8 | read only |
+
+## server\_version<a name="server_version"></a>
+
+Reports the version of PostgreSQL that this release of HAWQ is based on.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|---------------------|
+| string | 8.2.15 | read only |
+
+## server\_version\_num<a name="server_version_num"></a>
+
+Reports the version of PostgreSQL that this release of HAWQ is based on as an integer.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|---------------------|
+| integer | 80215 | read only |
+
+## shared\_buffers<a name="shared_buffers"></a>
+
+Sets the amount of memory a HAWQ segment instance uses for shared memory buffers. This setting must be at least 128KB and at least 16KB times [max\_connections](#max_connections).
+
+Each HAWQ segment instance calculates and attempts to allocate certain amount of shared memory based on the segment configuration. The value of `shared_buffers` is significant portion of this shared memory calculation, but is not all it. When setting `shared_buffers`, the values for the operating system parameters `SHMMAX` or `SHMALL` might also need to be adjusted.
+
+The operating system parameter `SHMMAX` specifies maximum size of a single shared memory allocation. The value of `SHMMAX` must be greater than this value:
+
+``` pre
+ shared_buffers + other_seg_shmem
+```
+
+The value of *other\_seg\_shmem* is the portion the HAWQ shared memory calculation that is not accounted for by the `shared_buffers` value. The *other\_seg\_shmem* value will vary based on the segment configuration.
+
+With the default HAWQ parameter values, the value for *other\_seg\_shmem* is approximately 111MB for HAWQ segments and approximately 79MB for the HAWQ master.
+
+The operating system parameter `SHMALL` specifies the maximum amount of shared memory on the host. The value of `SHMALL` must be greater than this value:
+
+``` pre
+ (num_instances_per_host * ( shared_buffers + other_seg_shmem )) + other_app_shared_mem
+```
+
+The value of *other\_app\_shared\_mem* is the amount of shared memory that is used by other applications and processes on the host.
+
+When shared memory allocation errors occur, possible ways to resolve shared memory allocation issues are to increase `SHMMAX` or `SHMALL`, or decrease `shared_buffers` or `max_connections`.
+
+See HAWQ installation documentation for information about the HAWQ values for the parameters `SHMMAX` and `SHMALL`.
+
+| Value Range | Default | Set Classifications |
+|----------------------------------------|---------|--------------------------|
+| integer > 16K \* *max\_connections* | 125MB | segment, system, restart |
+
+## shared\_preload\_libraries<a name="shared_preload_libraries"></a>
+
+A comma-separated list of shared libraries that are to be preloaded at server start. PostgreSQL procedural language libraries can be preloaded in this way, typically by using the syntax '`$libdir/plXXX`' where *XXX* is pgsql, perl, tcl, or python. By preloading a shared library, the library startup time is avoided when the library is first used. If a specified library is not found, the server will fail to start.
+
+| Value Range | Default | Set Classifications |
+|-------------------------------|--------------|------------------------|
+| string (comma-separated list) | no value set | local, system, restart |
+
+## ssl<a name="ssl"></a>
+
+Enables SSL connections.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | off | master, system, restart |
+
+## ssl\_ciphers<a name="ssl_ciphers"></a>
+
+Specifies a list of SSL ciphers that are allowed to be used on secure connections. See the openssl manual page for a list of supported ciphers.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| string | ALL | master, system, restart |
+
+## standard\_conforming\_strings<a name="standard_conforming_strings"></a>
+
+Reports whether ordinary string literals ('...') treat backslashes literally, as specified in the SQL standard. The value is currently always off, indicating that backslashes are treated as escapes. It is planned that this will change to on in a future release when string literal syntax changes to meet the standard. Applications may check this parameter to determine how string literals will be processed. The presence of this parameter can also be taken as an indication that the escape string syntax (E'...') is supported.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|---------------------|
+| Boolean | off | read only |
+
+## statement\_timeout<a name="statement_timeout"></a>
+
+Abort any statement that takes over the specified number of milliseconds. 0 turns off the limitation.
+
+| Value Range | Default | Set Classifications |
+|------------------------|---------|-------------------------|
+| number of milliseconds | 0 | master, session, reload |
+
+## superuser\_reserved\_connections<a name="superuser_reserved_connections"></a>
+
+Determines the number of connection slots that are reserved for HAWQ superusers.
+
+| Value Range | Default | Set Classifications |
+|---------------------------------|---------|------------------------|
+| integer < *max\_connections* | 3 | local, system, restart |
+
+## tcp\_keepalives\_count<a name="tcp_keepalives_count"></a>
+
+How many keepalives may be lost before the connection is considered dead. A value of 0 uses the system default. If TCP\_KEEPCNT is not supported, this parameter must be 0.
+
+| Value Range | Default | Set Classifications |
+|---------------------------|---------|------------------------|
+| number of lost keepalives | 0 | local, system, restart |
+
+## tcp\_keepalives\_idle<a name="tcp_keepalives_idle"></a>
+
+Number of seconds between sending keepalives on an otherwise idle connection. A value of 0 uses the system default. If TCP\_KEEPIDLE is not supported, this parameter must be 0.
+
+| Value Range | Default | Set Classifications |
+|-------------------|---------|------------------------|
+| number of seconds | 0 | local, system, restart |
+
+## tcp\_keepalives\_interval<a name="tcp_keepalives_interval"></a>
+
+How many seconds to wait for a response to a keepalive before retransmitting. A value of 0 uses the system default. If TCP\_KEEPINTVL is not supported, this parameter must be 0.
+
+| Value Range | Default | Set Classifications |
+|-------------------|---------|------------------------|
+| number of seconds | 0 | local, system, restart |
+
+## temp\_buffers<a name="temp_buffers"></a>
+
+Sets the maximum number of temporary buffers used by each database session. These are session-local buffers used only for access to temporary tables. The setting can be changed within individual sessions, but only up until the first use of temporary tables within a session. The cost of setting a large value in sessions that do not actually need a lot of temporary buffers is only a buffer descriptor, or about 64 bytes, per increment. However if a buffer is actually used, an additional 8192 bytes will be consumed.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| integer | 1024 | master, session, reload |
+
+## TimeZone<a name="TimeZone"></a>
+
+Sets the time zone for displaying and interpreting time stamps. The default is to use whatever the system environment specifies as the time zone. See [<span class="ph">Date/Time Keywords</span>](http://www.postgresql.org/docs/8.2/static/datetime-keywords.html) in the PostgreSQL documentation.
+
+| Value Range | Default | Set Classifications |
+|------------------------|---------|---------------------|
+| time zone abbreviation | | local, restart |
+
+## timezone\_abbreviations<a name="timezone_abbreviations"></a>
+
+Sets the collection of time zone abbreviations that will be accepted by the server for date time input. The default is `Default`, which is a collection that works in most of the world. `Australia` and `India`, and other collections can be defined for a particular installation. Possible values are names of configuration files stored in `$GPHOME/share/postgresql/timezonesets/`.
+
+To configure HAWQ to use a custom collection of timezones, copy the file that contains the timezone definitions to the directory `$GPHOME/share/postgresql/timezonesets/` on the HAWQ master and segment hosts. Then set value of the server configuration parameter `timezone_abbreviations` to the file. For example, to use a file `custom` that contains the default timezones and the WIB (Waktu Indonesia Barat) timezone.
+
+1. Copy the file `Default` from the directory `$GPHOME/share/postgresql/timezonesets/` the file `custom`. Add the WIB timezone information from the file `Asia.txt` to the `custom`.
+2. Copy the file `custom` to the directory `$GPHOME/share/postgresql/timezonesets/` on the HAWQ master and segment hosts.
+3. Set value of the server configuration parameter `timezone_abbreviations` to `custom`.
+4. Reload the server configuration file (`hawq stop -u`).
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| string | Default | master, session, reload |
+
+## track\_activities<a name="track_activities"></a>
+
+Enables the collection of statistics on the currently executing command of each session, along with the time at which that command began execution. When enabled, this information is not visible to all users, only to superusers and the user owning the session. This data can be accessed via the *pg\_stat\_activity* system view.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | on | master, session, reload |
+
+## track\_counts<a name="track_counts"></a>
+
+Enables the collection of row and block level statistics on database activity. If enabled, the data that is produced can be accessed via the *pg\_stat* and *pg\_statio* family of system views.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|------------------------|
+| Boolean | off | local, system, restart |
+
+## transaction\_isolation<a name="transaction_isolation"></a>
+
+Sets the current transaction's isolation level.
+
+<table>
+<colgroup>
+<col width="33%" />
+<col width="33%" />
+<col width="33%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Value Range</th>
+<th>Default</th>
+<th>Set Classifications</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>read committed
+<p>serializable</p></td>
+<td>read committed</td>
+<td>master, session, reload</td>
+</tr>
+</tbody>
+</table>
+
+## transaction\_read\_only<a name="transaction_read_only"></a>
+
+Sets the current transaction's read-only status.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | off | master, session, reload |
+
+## transform\_null\_equals<a name="transform_null_equals"></a>
+
+When on, expressions of the form expr = NULL (or NULL = expr) are treated as expr IS NULL, that is, they return true if expr evaluates to the null value, and false otherwise. The correct SQL-spec-compliant behavior of expr = NULL is to always return null (unknown).
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|-------------------------|
+| Boolean | off | master, session, reload |
+
+## unix\_socket\_directory<a name="unix_socket_directory"></a>
+
+Specifies the directory of the UNIX-domain socket on which the server is to listen for connections from client applications.
+
+| Value Range | Default | Set Classifications |
+|----------------|---------|------------------------|
+| directory path | unset | local, system, restart |
+
+## unix\_socket\_group<a name="unix_socket_group"></a>
+
+Sets the owning group of the UNIX-domain socket. By default this is an empty string, which uses the default group for the current user.
+
+| Value Range | Default | Set Classifications |
+|-----------------|---------|------------------------|
+| UNIX group name | unset | local, system, restart |
+
+## unix\_socket\_permissions<a name="unix_socket_permissions"></a>
+
+Sets the access permissions of the UNIX-domain socket. UNIX-domain sockets use the usual UNIX file system permission set. Note that for a UNIX-domain socket, only write permission matters.
+
+| Value Range | Default | Set Classifications |
+|------------------------------------------------------------------------------------|---------|------------------------|
+| numeric UNIX file permission mode (as accepted by the *chmod* or *umask* commands) | 511 | local, system, restart |
+
+## update\_process\_title<a name="update_process_title"></a>
+
+Enables updating of the process title every time a new SQL command is received by the server. The process title is typically viewed by the `ps` command.
+
+| Value Range | Default | Set Classifications |
+|-------------|---------|------------------------|
+| Boolean | on | local, system, restart |
+
+## vacuum\_cost\_delay<a name="vacuum_cost_delay"></a>
+
+The length of time that the process will sleep when the cost limit has been exceeded. 0 disables the cost-based vacuum delay feature.
+
+| Value Range | Default | Set Classifications |
+|------------------------------------------|---------|------------------------|
+| milliseconds < 0 (in multiples of 10) | 0 | local, system, restart |
+
+## vacuum\_cost\_limit<a name="vacuum_cost_limit"></a>
+
+The accumulated cost that will cause the vacuuming process to sleep.
+
+| Value Range | Default | Set Classifications |
+|----------------|---------|------------------------|
+| integer > 0 | 200 | local, system, restart |
+
+## vacuum\_cost\_page\_dirty<a name="vacuum_cost_page_dirty"></a>
+
+The estimated cost charged when vacuum modifies a block that was previously clean. It represents the extra I/O required to flush the dirty block out to disk again.
+
+| Value Range | Default | Set Classifications |
+|----------------|---------|------------------------|
+| integer > 0 | 20 | local, system, restart |
+
+## vacuum\_cost\_page\_miss<a name="vacuum_cost_page_miss"></a>
+
+The estimated cost for vacuuming a buffer that has to be read from disk. This represents the effort to lock the buffer pool, lookup the shared hash table, read the desired block in from the disk and scan its content.
+
+| Value Range | Default | Set Classifications |
+|----------------|---------|------------------------|
+| integer > 0 | 10 | local, system, restart |
+
+## vacuum\_freeze\_min\_age<a name="vacuum_freeze_min_age"></a>
+
+Specifies the cutoff age (in transactions) that `VACUUM` should use to decide whether to replace transaction IDs with *FrozenXID* while scanning a table.
+
+For information about `VACUUM` and transaction ID management, see [Managing Data](../../datamgmt/dml.html#topic1) and the [PostgreSQL documentation](http://www.postgresql.org/docs/8.2/static/routine-vacuuming.html#VACUUM-FOR-WRAPAROUND).
+
+| Value Range | Default | Set Classifications |
+|------------------------|-----------|------------------------|
+| integer 0-100000000000 | 100000000 | local, system, restart |
+
+## xid\_stop\_limit<a name="xid_stop_limit"></a>
+
+The number of transaction IDs prior to the ID where transaction ID wraparound occurs. When this limit is reached, HAWQ stops creating new transactions to avoid data loss due to transaction ID wraparound.
+
+| Value Range | Default | Set Classifications |
+|----------------|----------|------------------------|
+| integer > 0 | 20000000 | local, system, restart |
+
+
diff --git a/reference/hawq-reference.html.md.erb b/reference/hawq-reference.html.md.erb
new file mode 100644
index 0000000..f5abd2a
--- /dev/null
+++ b/reference/hawq-reference.html.md.erb
@@ -0,0 +1,43 @@
+---
+title: HAWQ Reference
+---
+
+This section provides a complete reference to HAWQ SQL commands, management utilities, configuration parameters, environment variables, and database objects.
+
+- **[Server Configuration Parameter Reference](../reference/HAWQSiteConfig.html)**
+
+ This section describes all server configuration guc/parameters that are available in HAWQ.
+
+- **[HDFS Configuration Reference](../reference/HDFSConfigurationParameterReference.html)**
+
+ This reference page describes HDFS configuration values that are configured for HAWQ either within `hdfs-site.xml`, `core-site.xml`, or `hdfs-client.xml`.
+
+- **[Environment Variables](../reference/HAWQEnvironmentVariables.html)**
+
+ This topic contains a reference of the environment variables that you set for HAWQ.
+
+- **[Character Set Support Reference](../reference/CharacterSetSupportReference.html)**
+
+ This topic provides a referene of the character sets supported in HAWQ.
+
+- **[Data Types](../reference/HAWQDataTypes.html)**
+
+ This topic provides a reference of the data types supported in HAWQ.
+
+- **[SQL Commands](../reference/SQLCommandReference.html)**
+
+ This section contains a description and the syntax of the SQL commands supported by HAWQ.
+
+- **[System Catalog Reference](../reference/catalog/catalog_ref.html)**
+
+ This reference describes the HAWQ system catalog tables and views.
+
+- **[The hawq\_toolkit Administrative Schema](../reference/toolkit/hawq_toolkit.html)**
+
+ This section provides a reference on the `hawq_toolkit` administrative schema.
+
+- **[HAWQ Management Tools Reference](../reference/cli/management_tools.html)**
+
+ Reference information for command-line utilities available in HAWQ.
+
+
diff --git a/reference/sql/ABORT.html.md.erb b/reference/sql/ABORT.html.md.erb
new file mode 100644
index 0000000..246b6eb
--- /dev/null
+++ b/reference/sql/ABORT.html.md.erb
@@ -0,0 +1,37 @@
+---
+title: ABORT
+---
+
+Aborts the current transaction.
+
+## Synopsis<a id="synop"></a>
+
+``` pre
+ABORT [WORK | TRANSACTION]
+```
+
+## Description<a id="abort__section3"></a>
+
+`ABORT` rolls back the current transaction and causes all the updates made by the transaction to be discarded. This command is identical in behavior to the standard SQL command `ROLLBACK`, and is present only for historical reasons.
+
+## Parameters<a id="abort__section4"></a>
+
+WORK
+TRANSACTION
+Optional key words. They have no effect.
+
+## Notes<a id="abort__section5"></a>
+
+Use `COMMIT` to successfully terminate a transaction.
+
+Issuing `ABORT` when not inside a transaction does no harm, but it will provoke a warning message.
+
+## Compatibility<a id="compat"></a>
+
+This command is a HAWQ extension present for historical reasons. ROLLBACK is the equivalent standard SQL command.
+
+## See Also<a id="see"></a>
+
+[BEGIN](BEGIN.html), [COMMIT](COMMIT.html), [ROLLBACK](ROLLBACK.html)
+
+
diff --git a/reference/sql/ALTER-AGGREGATE.html.md.erb b/reference/sql/ALTER-AGGREGATE.html.md.erb
new file mode 100644
index 0000000..9dc783b
--- /dev/null
+++ b/reference/sql/ALTER-AGGREGATE.html.md.erb
@@ -0,0 +1,69 @@
+---
+title: ALTER AGGREGATE
+---
+
+Changes the definition of an aggregate function.
+
+## Synopsis<a id="synop"></a>
+
+``` pre
+ALTER AGGREGATE name ( type [ , ... ] ) RENAME TO new_name
+
+ALTER AGGREGATE name ( type [ , ... ] ) OWNER TO new_owner
+
+ALTER AGGREGATE name ( type [ , ... ] ) SET SCHEMA new_schema
+
+```
+
+## Description<a id="desc"></a>
+
+`ALTER AGGREGATE` changes the definition of an aggregate function.
+
+You must own the aggregate function to use `ALTER AGGREGATE`. To change the schema of an aggregate function, you must also have `CREATE` privilege on the new schema. To alter the owner, you must also be a direct or indirect member of the new owning role, and that role must have `CREATE` privilege on the aggregate function’s schema. (These restrictions enforce that altering the owner does not do anything you could not do by dropping and recreating the aggregate function. However, a superuser can alter ownership of any aggregate function anyway.)
+
+## Parameters<a id="alteraggregate__section4"></a>
+
+ *name*
+The name (optionally schema-qualified) of an existing aggregate function.
+
+ *type*
+An input data type on which the aggregate function operates. To reference a zero-argument aggregate function, write \* in place of the list of input data types.
+
+ *new\_name*
+The new name of the aggregate function.
+
+ *new\_owner*
+The new owner of the aggregate function.
+
+ *new\_schema*
+The new schema for the aggregate function.
+
+## Examples<a id="alteraggregate__section5"></a>
+
+To rename the aggregate function `myavg` for type `integer` to `my_average`:
+
+``` pre
+ALTER AGGREGATE myavg(integer) RENAME TO my_average;
+```
+
+To change the owner of the aggregate function `myavg` for type `integer` to `joe`:
+
+``` pre
+ALTER AGGREGATE myavg(integer) OWNER TO joe;
+```
+
+To move the aggregate function `myavg` for type `integer` into schema `myschema`:
+
+``` pre
+ALTER AGGREGATE myavg(integer) SET SCHEMA myschema;
+```
+
+## Compatibility<a id="compat"></a>
+
+There is no `ALTER AGGREGATE` statement in the SQL standard.
+
+## See Also<a id="see"></a>
+
+[CREATE AGGREGATE](CREATE-AGGREGATE.html), [DROP AGGREGATE](DROP-AGGREGATE.html)
+
+
diff --git a/reference/sql/ALTER-DATABASE.html.md.erb b/reference/sql/ALTER-DATABASE.html.md.erb
new file mode 100644
index 0000000..a4ef760
--- /dev/null
+++ b/reference/sql/ALTER-DATABASE.html.md.erb
@@ -0,0 +1,52 @@
+---
+title: ALTER DATABASE
+---
+
+Changes the attributes of a database.
+
+## Synopsis<a id="alterrole__section2"></a>
+
+``` pre
+ALTER DATABASE name SET parameter { TO | + } { value | DEFAULT }
+
+ALTER DATABASE name RESET parameter
+```
+
+## Description<a id="desc"></a>
+
+`ALTER DATABASE` changes the attributes of a HAWQ database.
+
+`SET` and `RESET` *parameter* changes the session default for a configuration parameter for a HAWQ database. Whenever a new session is subsequently started in that database, the specified value becomes the session default value. The database-specific default overrides whatever setting is present in the server configuration file (`hawq-site.xml`). Only the database owner or a superuser can change the session defaults for a database. Certain parameters cannot be set this way, or can only be set by a superuser.
+
+## Parameters<a id="alterrole__section4"></a>
+
+ *name*
+The name of the database whose attributes are to be altered.
+
+**Note:** HAWQ reserves the database "hcatalog" for system use. You cannot connect to or alter the system "hcatalog" database.
+
+ *parameter*
+Set this database's session default for the specified configuration parameter to the given value. If value is `DEFAULT` or if `RESET` is used, the database-specific setting is removed, so the system-wide default setting will be inherited in new sessions. Use `RESET ALL` to clear all database-specific settings. See [About Server Configuration Parameters](../guc/guc_config.html#topic1) for information about user-settable configuration parameters.
+
+## Notes<a id="notes"></a>
+
+It is also possible to set a configuration parameter session default for a specific role (user) rather than to a database. Role-specific settings override database-specific ones if there is a conflict. See [ALTER ROLE](ALTER-ROLE.html).
+
+## Examples<a id="examples"></a>
+
+To set the default schema search path for the `mydatabase` database:
+
+``` pre
+ALTER DATABASE mydatabase SET search_path TO myschema,
+public, pg_catalog;
+```
+
+## Compatibility<a id="compat"></a>
+
+The `ALTER DATABASE` statement is a HAWQ extension.
+
+## See Also<a id="see"></a>
+
+[CREATE DATABASE](CREATE-DATABASE.html#topic1), [DROP DATABASE](DROP-DATABASE.html#topic1), [SET](SET.html)
+
+
diff --git a/reference/sql/ALTER-FUNCTION.html.md.erb b/reference/sql/ALTER-FUNCTION.html.md.erb
new file mode 100644
index 0000000..7ac8ffc
--- /dev/null
+++ b/reference/sql/ALTER-FUNCTION.html.md.erb
@@ -0,0 +1,108 @@
+---
+title: ALTER FUNCTION
+---
+
+Changes the definition of a function.
+
+## Synopsis<a id="alterfunction__section2"></a>
+
+``` pre
+ALTER FUNCTION name ( [ [argmode] [argname] argtype [, ...] ] )
+ action [, ... ] [RESTRICT]
+
+ALTER FUNCTION name ( [ [argmode] [argname] argtype [, ...] ] )
+ RENAME TO new_name
+
+ALTER FUNCTION name ( [ [argmode] [argname] argtype [, ...] ] )
+ OWNER TO new_owner
+
+ALTER FUNCTION name ( [ [argmode] [argname] argtype [, ...] ] )
+ SET SCHEMA new_schema
+
+```
+
+where *action* is one of:
+
+``` pre
+{CALLED ON NULL INPUT | RETURNS NULL ON NULL INPUT | STRICT}
+{IMMUTABLE | STABLE | VOLATILE}
+{[EXTERNAL] SECURITY INVOKER | [EXTERNAL] SECURITY DEFINER}
+```
+
+## Description<a id="desc"></a>
+
+`ALTER FUNCTION` changes the definition of a function.
+
+You must own the function to use `ALTER FUNCTION`. To change a function’s schema, you must also have `CREATE` privilege on the new schema. To alter the owner, you must also be a direct or indirect member of the new owning role, and that role must have `CREATE` privilege on the function’s schema. (These restrictions enforce that altering the owner does not do anything you could not do by dropping and recreating the function. However, a superuser can alter ownership of any function anyway.)
+
+## Parameters<a id="alterfunction__section4"></a>
+
+ *name*
+The name (optionally schema-qualified) of an existing function.
+
+ *argmode*
+The mode of an argument: either `IN`, `OUT`, or `INOUT`. If omitted, the default is `IN`. Note that `ALTER FUNCTION` does not actually pay any attention to `OUT` arguments, since only the input arguments are needed to determine the function's identity. So it is sufficient to list the `IN` and `INOUT` arguments.
+
+ *argname*
+The name of an argument. Note that `ALTER FUNCTION` does not actually pay any attention to argument names, since only the argument data types are needed to determine the function's identity.
+
+ *argtype*
+The data type(s) of the function's arguments (optionally schema-qualified), if any.
+
+ *new\_name*
+The new name of the function.
+
+ *new\_owner*
+The new owner of the function. Note that if the function is marked `SECURITY DEFINER`, it will subsequently execute as the new owner.
+
+ *new\_schema*
+The new schema for the function.
+
+CALLED ON NULL INPUT
+RETURNS NULL ON NULL INPUT
+STRICT
+`CALLED ON NULL INPUT` changes the function so that it will be invoked when some or all of its arguments are null. `RETURNS NULL ON NULL INPUT` or `STRICT` changes the function so that it is not invoked if any of its arguments are null; instead, a null result is assumed automatically. See `CREATE FUNCTION` for more information.
+
+IMMUTABLE
+STABLE
+VOLATILE,
+Change the volatility of the function to the specified setting. See `CREATE FUNCTION` for details.
+
+\[ EXTERNAL \] SECURITY INVOKER
+\[ EXTERNAL \] SECURITY DEFINER
+Change whether the function is a security definer or not. The key word `EXTERNAL` is ignored for SQL conformance. See `CREATE FUNCTION` for more information about this capability.
+
+RESTRICT
+Ignored for conformance with the SQL standard.
+
+## Notes<a id="notes"></a>
+
+HAWQ has limitations on the use of functions defined as `STABLE` or `VOLATILE`. See [CREATE FUNCTION](CREATE-FUNCTION.html) for more information.
+
+## Examples<a id="alterfunction__section6"></a>
+
+To rename the function `sqrt` for type `integer` to `square_root`:
+
+``` pre
+ALTER FUNCTION sqrt(integer) RENAME TO square_root;
+```
+
+To change the owner of the function `sqrt` for type `integer` to `joe`:
+
+``` pre
+ALTER FUNCTION sqrt(integer) OWNER TO joe;
+```
+
+To change the *schema* of the function `sqrt` for type `integer` to `math`:
+
+``` pre
+ALTER FUNCTION sqrt(integer) SET SCHEMA math;
+```
+
+## Compatibility<a id="compat"></a>
+
+This statement is partially compatible with the `ALTER FUNCTION` statement in the SQL standard. The standard allows more properties of a function to be modified, but does not provide the ability to rename a function, make a function a security definer, or change the owner, schema, or volatility of a function. The standard also requires the `RESTRICT` key word, which is optional in HAWQ.
+
+## See Also<a id="see"></a>
+
+[CREATE AGGREGATE](CREATE-AGGREGATE.html), [DROP AGGREGATE](DROP-AGGREGATE.html)
diff --git a/reference/sql/ALTER-OPERATOR-CLASS.html.md.erb b/reference/sql/ALTER-OPERATOR-CLASS.html.md.erb
new file mode 100644
index 0000000..42d18b5
--- /dev/null
+++ b/reference/sql/ALTER-OPERATOR-CLASS.html.md.erb
@@ -0,0 +1,44 @@
+---
+title: ALTER OPERATOR CLASS
+---
+
+Changes the definition of an operator class.
+
+## Synopsis<a id="synop"></a>
+
+``` pre
+ALTER OPERATOR CLASS name USING index_method RENAME TO newname
+
+ALTER OPERATOR CLASS name USING index_method OWNER TO newowner
+
+```
+
+## Description<a id="desc"></a>
+
+`ALTER OPERATOR CLASS` changes the definition of an operator class.
+
+You must own the operator class to use `ALTER OPERATOR CLASS`. To alter the owner, you must also be a direct or indirect member of the new owning role, and that role must have `CREATE` privilege on the operator class’s schema. (These restrictions enforce that altering the owner does not do anything you could not do by dropping and recreating the operator class. However, a superuser can alter ownership of any operator class anyway.)
+
+## Parameters<a id="alteroperatorclass__section4"></a>
+
+ *name*
+The name (optionally schema-qualified) of an existing operator class.
+
+ *index\_method*
+The name of the index method this operator class is for.
+
+ *newname*
+The new name of the operator class.
+
+ *newowner*
+The new owner of the operator class
+
+## Compatibility<a id="compat"></a>
+
+There is no `ALTER OPERATOR` statement in the SQL standard.
+
+## See Also<a id="see"></a>
+
+[CREATE OPERATOR](CREATE-OPERATOR.html), [DROP OPERATOR CLASS](DROP-OPERATOR-CLASS.html)
+
+
diff --git a/reference/sql/ALTER-OPERATOR.html.md.erb b/reference/sql/ALTER-OPERATOR.html.md.erb
new file mode 100644
index 0000000..10905ee
--- /dev/null
+++ b/reference/sql/ALTER-OPERATOR.html.md.erb
@@ -0,0 +1,51 @@
+---
+title: ALTER OPERATOR
+---
+
+Changes the definition of an operator.
+
+## Synopsis<a id="synop"></a>
+
+``` pre
+ALTER OPERATOR name ( {lefttype | NONE} , {righttype | NONE} )
+ OWNER TO newowner
+
+```
+
+## Description<a id="desc"></a>
+
+`ALTER OPERATOR` changes the definition of an operator. The only currently available functionality is to change the owner of the operator.
+
+You must own the operator to use `ALTER OPERATOR`. To alter the owner, you must also be a direct or indirect member of the new owning role, and that role must have `CREATE` privilege on the operator’s schema. (These restrictions enforce that altering the owner does not do anything you could not do by dropping and recreating the operator. However, a superuser can alter ownership of any operator anyway.)
+
+## Parameters<a id="alteroperator__section4"></a>
+
+ *name*
+The name (optionally schema-qualified) of an existing operator.
+
+ *lefttype*
+The data type of the operator's left operand; write `NONE` if the operator has no left operand.
+
+ *righttype*
+The data type of the operator's right operand; write `NONE` if the operator has no right operand.
+
+ *newowner*
+The new owner of the operator.
+
+## Example<a id="example"></a>
+
+Change the owner of a custom operator `a @@ b` for type `text`:
+
+``` pre
+ALTER OPERATOR @@ (text, text) OWNER TO joe;
+```
+
+## Compatibility<a id="compat"></a>
+
+There is no `ALTER OPERATOR` statement in the SQL standard.
+
+## See Also<a id="see"></a>
+
+[CREATE OPERATOR](CREATE-OPERATOR.html), [DROP OPERATOR](DROP-OPERATOR.html)
+
+
diff --git a/reference/sql/ALTER-RESOURCE-QUEUE.html.md.erb b/reference/sql/ALTER-RESOURCE-QUEUE.html.md.erb
new file mode 100644
index 0000000..10790ce
--- /dev/null
+++ b/reference/sql/ALTER-RESOURCE-QUEUE.html.md.erb
@@ -0,0 +1,131 @@
+---
+title: ALTER RESOURCE QUEUE
+---
+
+Modify an existing resource queue.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+ALTER RESOURCE QUEUE name WITH (queue_attribute=value [, ... ])
+```
+
+where *queue\_attribute* is:
+
+``` pre
+[ACTIVE_STATEMENTS=integer]
+ MEMORY_LIMIT_CLUSTER=percentage
+ CORE_LIMIT_CLUSTER=percentage
+ [ALLOCATION_POLICY='even']
+ [VSEGMENT_RESOURCE_QUOTA='mem:memory_units']
+ [RESOURCE_OVERCOMMIT_FACTOR=double]
+ [NVSEG_UPPER_LIMIT=integer]
+ [NVSEG_LOWER_LIMIT=integer]
+ [NVSEG_UPPER_LIMIT_PERSEG=double]
+ [NVSEG_LOWER_LIMIT_PERSEG=double]
+
+ memory_units ::= {128mb|256mb|512mb|1024mb|2048mb|4096mb|
+ 8192mb|16384mb|1gb|2gb|4gb|8gb|16gb}
+ percentage ::= integer %
+```
+
+## Description<a id="topic1__section3"></a>
+
+Changes attributes for an existing resource queue in HAWQ. You cannot change the parent of an existing resource queue, and you cannot change a resource queue while it is active. Only a superuser can modify a resource queue.
+
+Resource queues with an `ACTIVE_STATEMENTS` threshold set a maximum limit on the number parallel active query statements that can be executed by roles assigned to tha leaf queue. It controls the number of active queries that are allowed to run at the same time. The value for `ACTIVE_STATEMENTS` should be an integer greater than 0. If not specified, the default value is 20.
+
+When modifying the resource queue, use the MEMORY\_LIMIT\_CLUSTER and CORE\_LIMIT\_CLUSTER to tune the allowed resource usage of the resource queue. MEMORY\_LIMIT\_CLUSTER and CORE\_LIMIT\_CLUSTER must be equal for the same resource queue. In addition the sum of the percentages of MEMORY\_LIMIT\_CLUSTER (and CORE\_LIMIT\_CLUSTER) for resource queues that share the same parent cannot exceed 100%.
+
+To modify the role associated with the resource queue, use the [ALTER ROLE](ALTER-ROLE.html) or [CREATE ROLE](CREATE-ROLE.html) command. You can only assign roles to the leaf-level resource queues (resource queues that do not have any children.)
+
+The default memory allotment can be overridden on a per-query basis by using `hawq_rm_stmt_vseg_memory` and` hawq_rm_stmt_nvseg` configuration parameters. See [Configuring Resource Quotas for Query Statements](/20/resourcemgmt/ConfigureResourceManagement.html#topic_g2p_zdq_15).
+
+To see the status of a resource queue, see [Checking Existing Resource Queues](/20/resourcemgmt/ResourceQueues.html#topic_lqy_gls_zt).
+
+See also [Best Practices for Using Resource Queues](../../bestpractices/managing_resources_bestpractices.html#topic_hvd_pls_wv).
+
+## Parameters<a id="topic1__section4"></a>
+
+ *name*
+Required. The name of the resource queue you wish to modify.
+
+<!-- -->
+
+ACTIVE\_STATEMENTS *integer*
+Optional. Defines the limit of the number of parallel active statements in one leaf queue. The maximum number of connections cannot exceed this limit. If this limit is reached, the HAWQ resource manager queues more query allocation requests. Note that a single session can have several concurrent statement executions that occupy multiple connection resources. The value for `ACTIVE_STATEMENTS` should be an integer greater than 0. The default value is 20.
+
+MEMORY\_LIMIT\_CLUSTER *percentage*
+Required. Defines how much memory a resource queue can consume from its parent resource queue and consequently dispatch to the execution of parallel statements. The valid values are 1% to 100%. The value of MEMORY\_ LIMIT\_CLUSTER must be identical to the value of CORE\_LIMIT\_CLUSTER. The sum of values for MEMORY\_LIMIT\_CLUSTER of this queue plus other queues that share the same parent cannot exceed 100%. The HAWQ resource manager periodically validates this restriction.
+
+**Note:** If you want to increase the percentage, you may need to decrease the percentage of any resource queue(s) that share the same parent resource queue first. The total cannot exceed 100%.
+
+CORE\_LIMIT\_CLUSTER *percentage*
+Required. The percentage of consumable CPU (virtual core) resources that the resource queue can take from its parent resource queue. The valid values are 1% to 100%. The value of CORE\_ LIMIT\_CLUSTER must be identical to the value of MEMORY\_LIMIT\_CLUSTER. The sum of values for CORE\_LIMIT\_CLUSTER of this queue and queues that share the same parent cannot exceed 100%.
+
+**Note:** If you want to increase the percentage, you may need to decrease the percentage of any resource queue(s) that share the same parent resource queue first. The total cannot exceed 100%.
+
+ALLOCATION\_POLICY *string*
+Optional. Defines the resource allocation policy for parallel statement execution. The default value is `even`.
+
+**Note:** This release only supports an `even` allocation policy. Even if you do not specify this attribute, the resource queue still applies an `even` allocation policy. Future releases will support alternative allocation policies.
+
+Setting the allocation policy to `even` means resources are always evenly dispatched based on current concurrency. When multiple query resource allocation requests are queued, the resource queue tries to evenly dispatch resources to queued requests until one of the following conditions are encountered:
+
+- There are no more allocated resources in this queue to dispatch, or
+- The ACTIVE\_STATEMENTS limit has been reached
+
+For each query resource allocation request, the HAWQ resource mananger determines the minimum and maximum size of a virtual segment based on multiple factors including query cost, user configuration, table properties, and so on. For example, a hash distributed table requires fixed size of virtual segments. With an even allocation policy, the HAWQ resource manager uses the minimum virtual segment size requirement and evenly dispatches resources to each query resource allocation request in the resource queue.
+
+VSEG\_RESOURCE\_QUOTA 'mem:{128mb | 256mb | 512mb | 1024mb | 2048mb | 4096mb | 8192mb | 16384mb | 1gb | 2gb | 4gb | 8gb | 16gb}'
+Optional. This quota defines how resources are split across multiple virtual segments. For example, when the HAWQ resource manager determines that 256GB memory and 128 vcores should be allocated to the current resource queue, there are multiple solutions on how to divide the resources across virtual segments. For example, you could use a) 2GB/1 vcore \* 128 virtual segments or b) 1GB/0.5 vcore \* 256 virtual segments. Therefore, you can use this attribute to make the HAWQ resource manager calculate the number of virtual segments based on how to divide the memory. For example, if `VSEGMENT_RESOURCE_QUOTA``='mem:512mb'`, then the resource queue will use 512MB/0.25 vcore \* 512 virtual segments. The default value is '`mem:256mb`'.
+
+**Note:** To avoid resource fragmentation, make sure that the segment resource capacity configured for HAWQ (in HAWQ Standalone mode: `hawq_rm_memory_limit_perseg`; in YARN mode: `yarn.nodemanager.resource.memory-mb` must be a multiple of the resource quotas for all virtual segments and CPU to memory ratio must be a multiple of the amount configured for `yarn.scheduler.minimum-allocation-mb`.
+
+RESOURCE\_OVERCOMMIT\_FACTOR *double*
+Optional. This factor defines how much a resource can be overcommitted. The default value is `2.0`. For example, if RESOURCE\_OVERCOMMIT\_FACTOR is set to 3.0 and MEMORY\_LIMIT\_CLUSTER is set to 30%, then the maximum possible resource allocation in this queue is 90% (30% x 3.0). If the resulting maximum is bigger than 100%, then 100% is adopted. The minimum value that this attribute can be set to is `1.0`.
+
+NVSEG\_UPPER\_LIMIT *integer* / NVSEG\_UPPER\_LIMIT\_PERSEG *double*
+Optional. These limits restrict the range of number of virtual segments allocated in this resource queue for executing one query statement. NVSEG\_UPPER\_LIMIT defines an upper limit of virtual segments for one statement execution regardless of actual cluster size, while NVSEG\_UPPER\_LIMIT\_PERSEG defines the same limit by using the average number of virtual segments in one physical segment. Therefore, the limit defined by NVSEG\_UPPER\_LIMIT\_PERSEG varies dynamically according to the changing size of the HAWQ cluster.
+
+For example, if you set `NVSEG_UPPER_LIMIT=10` all query resource requests are strictly allocated no more than 10 virtual segments. If you set NVSEG\_UPPER\_LIMIT\_PERSEG=2 and assume that currently there are 5 available HAWQ segments in the cluster, query resource requests are allocated 10 virtual segments at the most.
+
+NVSEG\_UPPER\_LIMIT cannot be set to a lower value than NVSEG\_LOWER\_LIMIT if both limits are enabled. In addition, the upper limit cannot be set to a value larger than the value set in global configuration parameter `hawq_rm_nvseg_perquery_limit` and `hawq_rm_nvseg_perquery_perseg_limit`.
+
+By default, both limits are set to **-1**, which means the limits are disabled. `NVSEG_UPPER_LIMIT` has higher priority than `NVSEG_UPPER_LIMIT_PERSEG`. If both limits are set, then `NVSEG_UPPER_LIMIT_PERSEG` is ignored. If you have enabled resource quotas for the query statement, then these limits are ignored.
+
+**Note:** If the actual lower limit of the number of virtual segments becomes greater than the upper limit, then the lower limit is automatically reduced to be equal to the upper limit. This situation is possible when user sets both `NVSEG_UPPER_LIMIT `and `NVSEG_LOWER_LIMIT_PERSEG`. After expanding the cluster, the dynamic lower limit may become greater than the value set for the fixed upper limit.
+
+NVSEG\_LOWER\_LIMIT *integer* / NVSEG\_LOWER\_LIMIT\_PERSEG *double*
+Optional. These limits specify the minimum number of virtual segments allocated for one statement execution in order to guarantee query performance. NVSEG\_LOWER\_LIMIT defines the lower limit of virtual segments for one statement execution regardless the actual cluster size, while NVSEG\_LOWER\_LIMIT\_PERSEG defines the same limit by the average virtual segment number in one segment. Therefore, the limit defined by NVSEG\_LOWER\_LIMIT\_PERSEG varies dynamically along with the size of HAWQ cluster.
+
+NVSEG\_UPPER\_LIMIT\_PERSEG cannot be less than NVSEG\_LOWER\_LIMIT\_PERSEG if both limits are set enabled.
+
+For example, if you set NVSEG\_LOWER\_LIMIT=10, and one statement execution potentially needs no fewer than 10 virtual segments, then this request has at least 10 virtual segments allocated. If you set NVSEG\_UPPER\_LIMIT\_PERSEG=2, assuming there are currently 5 available HAWQ segments in the cluster, and one statement execution potentially needs no fewer than 10 virtual segments, then the query resource request will be allocated at least 10 virtual segments. If one statement execution needs at most 4 virtual segments, the resource manager will allocate at most 4 virtual segments instead of 10 since this resource request does not need more than 9 virtual segments.
+
+By default, both limits are set to **-1**, which means the limits are disabled. `NVSEG_LOWER_LIMIT` has higher priority than `NVSEG_LOWER_LIMIT_PERSEG`. If both limits are set, then `NVSEG_LOWER_LIMIT_PERSEG` is ignored. If you have enabled resource quotas for the query statement, then these limits are ignored.
+
+**Note:** If the actual lower limit of the number of virtual segments becomes greater than the upper limit, then the lower limit is automatically reduced to be equal to the upper limit. This situation is possible when user sets both `NVSEG_UPPER_LIMIT `and `NVSEG_LOWER_LIMIT_PERSEG`. After expanding the cluster, the dynamic lower limit may become greater than the value set for the fixed upper limit. .
+
+## Examples<a id="topic1__section6"></a>
+
+Change the memory and core limit of a resource queue:
+
+``` pre
+ALTER RESOURCE QUEUE test_queue_1 WITH (MEMORY_LIMIT_CLUSTER=40%,
+CORE_LIMIT_CLUSTER=40%);
+```
+
+Change the active statements maximum for the resource queue:
+
+``` pre
+ALTER RESOURCE QUEUE test_queue_1 WITH (ACTIVE_STATEMENTS=50);
+```
+
+## Compatibility<a id="topic1__section7"></a>
+
+`ALTER RESOURCE QUEUE` is a HAWQ extension. There is no provision for resource queues or workload management in the SQL standard.
+
+## See Also<a id="topic1__section8"></a>
+
+[ALTER ROLE](ALTER-ROLE.html), [CREATE RESOURCE QUEUE](CREATE-RESOURCE-QUEUE.html), [CREATE ROLE](CREATE-ROLE.html), [DROP RESOURCE QUEUE](DROP-RESOURCE-QUEUE.html)
diff --git a/reference/sql/ALTER-ROLE.html.md.erb b/reference/sql/ALTER-ROLE.html.md.erb
new file mode 100644
index 0000000..f88a5b5
--- /dev/null
+++ b/reference/sql/ALTER-ROLE.html.md.erb
@@ -0,0 +1,181 @@
+---
+title: ALTER ROLE
+---
+
+Changes a database role (user or group).
+
+## Synopsis<a id="alterrole__section2"></a>
+
+``` pre
+ALTER ROLE name RENAME TO newname
+
+ALTER ROLE name RESET config_parameter
+
+ALTER ROLE name RESOURCE QUEUE {queue_name | NONE}
+
+ALTER ROLE name [ [WITH] option [ ... ] ]
+```
+
+where *option* can be:
+
+``` pre
+ SUPERUSER | NOSUPERUSER
+ | CREATEDB | NOCREATEDB
+ | CREATEROLE | NOCREATEROLE
+ | CREATEEXTTABLE | NOCREATEEXTTABLE
+ [ ( attribute='value'[, ...] ) ]
+ where attributes and value are:
+ type='readable'|'writable'
+ protocol='gpfdist'|'http'
+
+ | INHERIT | NOINHERIT
+ | LOGIN | NOLOGIN
+ | CONNECTION LIMIT connlimit
+ | [ENCRYPTED | UNENCRYPTED] PASSWORD 'password'
+ | VALID UNTIL 'timestamp'
+ | [ DENY deny_point ]
+ | [ DENY BETWEEN deny_point AND deny_point]
+ | [ DROP DENY FOR deny_point ]
+```
+
+## Description<a id="desc"></a>
+
+`ALTER ROLE` changes the attributes of a HAWQ role. There are several variants of this command:
+
+- **RENAME** — Changes the name of the role. Database superusers can rename any role. Roles having `CREATEROLE` privilege can rename non-superuser roles. The current session user cannot be renamed (connect as a different user to rename a role). Because MD5-encrypted passwords use the role name as cryptographic salt, renaming a role clears its password if the password is MD5-encrypted.
+- **SET | RESET** — changes a role’s session default for a specified configuration parameter. Whenever the role subsequently starts a new session, the specified value becomes the session default, overriding whatever setting is present in server configuration file (`hawq-site.xml`). For a role without LOGIN privilege, session defaults have no effect. Ordinary roles can change their own session defaults. Superusers can change anyone’s session defaults. Roles having `CREATEROLE` privilege can change defaults for non-superuser roles. See "Server Configuration Parameters" for more information on all user-settable configuration parameters.
+- **RESOURCE QUEUE** — Assigns the role to a workload management resource queue. The role would then be subject to the limits assigned to the resource queue when issuing queries. Specify `NONE` to assign the role to the default resource queue. A role can only belong to one resource queue. For a role without `LOGIN` privilege, resource queues have no effect. See [CREATE RESOURCE QUEUE](CREATE-RESOURCE-QUEUE.html#topic1) for more information.
+- **WITH** *option* — Changes many of the role attributes that can be specified in [CREATE ROLE](CREATE-ROLE.html). Attributes not mentioned in the command retain their previous settings. Database superusers can change any of these settings for any role. Roles having `CREATEROLE` privilege can change any of these settings, but only for non-superuser roles. Ordinary roles can only change their own password.
+
+## Parameters<a id="alterrole__section4"></a>
+
+ *name*
+The name of the role whose attributes are to be altered.
+
+ *newname*
+The new name of the role.
+
+ *config\_parameter=value*
+Set this role's session default for the specified configuration parameter to the given value. If value is `DEFAULT` or if `RESET` is used, the role-specific variable setting is removed, so the role will inherit the system-wide default setting in new sessions. Use `RESET ALL` to clear all role-specific settings. See [SET](SET.html) and [About Server Configuration Parameters](../guc/guc_config.html#topic1) for information about user-settable configuration parameters.
+
+ *queue\_name*
+The name of the resource queue to which the user-level role is to be assigned. Only roles with `LOGIN` privilege can be assigned to a resource queue. To unassign a role from a resource queue and put it in the default resource queue, specify `NONE`. A role can only belong to one resource queue.
+
+SUPERUSER | NOSUPERUSER
+CREATEDB | NOCREATEDB
+CREATEROLE | NOCREATEROLE
+CREATEEXTTABLE | NOCREATEEXTTABLE \[(*attribute*='*value*')\]
+If `CREATEEXTTABLE` is specified, the role being defined is allowed to create external tables. The default `type` is `readable` and the default `protocol` is `gpfdist` if not specified. `NOCREATEEXTTABLE` (the default) denies the role the ability to create external tables. Using the `file` protocol when creating external tables is not supported. This is because HAWQ cannot guarantee scheduling executors on a specific host. Likewise, you cannot use the `execute` command with `ON ALL` and `ON HOST` for the same reason. Use the `ON MASTER/number/SEGMENT segment_id` to specify which segment instances are to execute the command.
+
+INHERIT | NOINHERIT
+LOGIN | NOLOGIN
+CONNECTION LIMIT *connlimit*
+PASSWORD '*password*'
+ENCRYPTED | UNENCRYPTED
+VALID UNTIL '*timestamp*'
+These clauses alter role attributes originally set by [CREATE ROLE](CREATE-ROLE.html).
+
+DENY *deny\_point*
+DENY BETWEEN *deny\_point* AND *deny\_point*
+The `DENY` and `DENY BETWEEN` keywords set time-based constraints that are enforced at login. `DENY`sets a day or a day and time to deny access. `DENY BETWEEN` sets an interval during which access is denied. Both use the parameter *deny\_point* that has following format:
+
+``` pre
+DAY day [ TIME 'time' ]
+```
+
+The two parts of the `deny_point` parameter use the following formats:
+
+For *day*:
+
+``` pre
+{'Sunday' | 'Monday' | 'Tuesday' |'Wednesday' | 'Thursday' | 'Friday' |
+'Saturday' | 0-6 }
+```
+
+For *time*:
+
+``` pre
+{ 00-23 : 00-59 | 01-12 : 00-59 { AM | PM }}
+```
+
+The `DENY BETWEEN` clause uses two *deny\_point* parameters.
+
+``` pre
+DENY BETWEEN deny_point AND deny_point
+
+```
+
+DROP DENY FOR *deny\_point*
+The `DROP DENY FOR` clause removes a time-based constraint from the role. It uses the *deny\_point* parameter described above.
+
+## Notes
+
+Use `GRANT` and `REVOKE` for adding and removing role memberships.
+
+Caution must be exercised when specifying an unencrypted password with this command. The password will be transmitted to the server in clear text, and it might also be logged in the client’s command history or the server log. The `psql` command-line client contains a meta-command `\password` that can be used to safely change a role’s password.
+
+It is also possible to tie a session default to a specific database rather than to a role. Role-specific settings override database-specific ones if there is a conflict.
+
+## Examples
+
+Change the password for a role:
+
+``` pre
+ALTER ROLE daria WITH PASSWORD 'passwd123';
+```
+
+Change a password expiration date:
+
+``` pre
+ALTER ROLE scott VALID UNTIL 'May 4 12:00:00 2015 +1';
+```
+
+Make a password valid forever:
+
+``` pre
+ALTER ROLE luke VALID UNTIL 'infinity';
+```
+
+Give a role the ability to create other roles and new databases:
+
+``` pre
+ALTER ROLE joelle CREATEROLE CREATEDB;
+```
+
+Give a role a non-default setting of the `maintenance_work_mem` parameter:
+
+``` pre
+ALTER ROLE admin SET maintenance_work_mem = 100000;
+```
+
+Assign a role to a resource queue:
+
+``` pre
+ALTER ROLE sammy RESOURCE QUEUE poweruser;
+```
+
+Give a role permission to create writable external tables:
+
+``` pre
+ALTER ROLE load CREATEEXTTABLE (type='writable');
+```
+
+Alter a role so it does not allow login access on Sundays:
+
+``` pre
+ALTER ROLE user3 DENY DAY 'Sunday';
+```
+
+Alter a role to remove the constraint that does not allow login access on Sundays:
+
+``` pre
+ALTER ROLE user3 DROP DENY FOR DAY 'Sunday';
+```
+
+## Compatibility<a id="compat"></a>
+
+The `ALTER ROLE` statement is a HAWQ extension.
+
+## See Also<a id="see"></a>
+
+[CREATE ROLE](CREATE-ROLE.html), [DROP ROLE](DROP-ROLE.html), [SET](SET.html), [CREATE RESOURCE QUEUE](CREATE-RESOURCE-QUEUE.html), [GRANT](GRANT.html), [REVOKE](REVOKE.html)
diff --git a/reference/sql/ALTER-TABLE.html.md.erb b/reference/sql/ALTER-TABLE.html.md.erb
new file mode 100644
index 0000000..6e3532c
--- /dev/null
+++ b/reference/sql/ALTER-TABLE.html.md.erb
@@ -0,0 +1,421 @@
+---
+title: ALTER TABLE
+---
+
+Changes the definition of a table.
+
+## Synopsis<a id="altertable__section2"></a>
+
+``` pre
+ALTER TABLE [ONLY] name RENAME [COLUMN] column TO new_column
+
+ALTER TABLE name RENAME TO new_name
+
+ALTER TABLE name SET SCHEMA new_schema
+
+ALTER TABLE [ONLY] name SET
+ DISTRIBUTED BY (column, [ ... ] )
+ | DISTRIBUTED RANDOMLY
+ | WITH (REORGANIZE=true|false)
+
+ALTER TABLE [ONLY] name
+ action [, ... ]
+
+ALTER TABLE name
+ [ ALTER PARTITION { partition_name | FOR (RANK(number))
+ | FOR (value) } partition_action [...] ]
+ partition_action
+
+```
+
+where *action* is one of:
+
+``` pre
+ ADD [COLUMN] column_name type
+ [ ENCODING ( storage_directive [,…] ) ]
+ [column_constraint [ ... ]]
+ DROP [COLUMN] column [RESTRICT | CASCADE]
+ ALTER [COLUMN] column TYPE type [USING expression]
+ ALTER [COLUMN] column SET DEFAULT expression
+ ALTER [COLUMN] column DROP DEFAULT
+ ALTER [COLUMN] column { SET | DROP } NOT NULL
+ ALTER [COLUMN] column SET STATISTICS integer
+ ADD table_constraint
+ DROP CONSTRAINT constraint_name [RESTRICT | CASCADE]
+ SET WITHOUT OIDS
+ INHERIT parent_table
+ NO INHERIT parent_table
+ OWNER TO new_owner
+
+```
+
+where *partition\_action* is one of:
+
+``` pre
+ ALTER DEFAULT PARTITION
+ DROP DEFAULT PARTITION [IF EXISTS]
+ DROP PARTITION [IF EXISTS] { partition_name |
+ FOR (RANK(number)) | FOR (value) } [CASCADE]
+ TRUNCATE DEFAULT PARTITION
+ TRUNCATE PARTITION { partition_name | FOR (RANK(number)) |
+ FOR (value) }
+ RENAME DEFAULT PARTITION TO new_partition_name
+ RENAME PARTITION { partition_name | FOR (RANK(number)) |
+ FOR (value) } TO new_partition_name
+ ADD DEFAULT PARTITION name [ ( subpartition_spec ) ]
+ ADD PARTITION name
+ partition_element
+ [ ( subpartition_spec ) ]
+ EXCHANGE DEFAULT PARTITION WITH TABLE table_name
+ [ WITH | WITHOUT VALIDATION ]
+ EXCHANGE PARTITION { partition_name | FOR (RANK(number)) |
+ FOR (value) } WITH TABLE table_name
+ [ WITH | WITHOUT VALIDATION ]
+ SET SUBPARTITION TEMPLATE (subpartition_spec)
+ SPLIT DEFAULT PARTITION
+ { AT (list_value)
+ | START([datatype] range_value) [INCLUSIVE | EXCLUSIVE]
+ END([datatype] range_value) [INCLUSIVE | EXCLUSIVE] }
+ [ INTO ( PARTITION new_partition_name,
+ PARTITION default_partition_name ) ]
+ SPLIT PARTITION { partition_name | FOR (RANK(number)) |
+ FOR (value) } AT (value)
+ [ INTO (PARTITION partition_name, PARTITION partition_name)]
+```
+
+where *partition\_element* is:
+
+``` pre
+ VALUES (list_value [,...] )
+ | START ([datatype] 'start_value') [INCLUSIVE | EXCLUSIVE]
+ [ END ([datatype] 'end_value') [INCLUSIVE | EXCLUSIVE] ]
+ | END ([datatype] 'end_value') [INCLUSIVE | EXCLUSIVE]
+[ WITH ( partition_storage_parameter=value [, ... ] ) ]
+[ TABLESPACE tablespace ]
+```
+
+where *subpartition\_spec* is:
+
+``` pre
+ subpartition_element [, ...]
+```
+
+and *subpartition\_element* is:
+
+``` pre
+ DEFAULT SUBPARTITION subpartition_name
+ | [SUBPARTITION subpartition_name] VALUES (list_value [,...] )
+ | [SUBPARTITION subpartition_name]
+ START ([datatype] 'start_value') [INCLUSIVE | EXCLUSIVE]
+ [ END ([datatype] 'end_value') [INCLUSIVE | EXCLUSIVE] ]
+ [ EVERY ( [number | datatype] 'interval_value') ]
+ | [SUBPARTITION subpartition_name]
+ END ([datatype] 'end_value') [INCLUSIVE | EXCLUSIVE]
+ [ EVERY ( [number | datatype] 'interval_value') ]
+[ WITH ( partition_storage_parameter=value [, ... ] ) ]
+[ TABLESPACE tablespace ]
+```
+
+where *storage\_parameter* is:
+
+``` pre
+ APPENDONLY={TRUE}
+ BLOCKSIZE={8192-2097152}
+ ORIENTATION={ROW | PARQUET}
+ COMPRESSTYPE={ZLIB|SNAPPY|GZIP|NONE}
+ COMPRESSLEVEL={0-9}
+ FILLFACTOR={10-100}
+ OIDS[=TRUE|FALSE]
+```
+
+where *storage\_directive* is:
+
+``` pre
+ COMPRESSTYPE={ZLIB|SNAPPY|GZIP|NONE}}
+ | COMPRESSLEVEL={0-9}
+ | BLOCKSIZE={8192-2097152}
+```
+
+where *column\_reference\_storage\_directive* is:
+
+``` pre
+ COLUMN column_name ENCODING ( storage_directive [, … ] ), …
+ | DEFAULT COLUMN ENCODING ( storage_directive [, … ] )
+```
+
+**Note:**
+When using multi-level partition designs, the following operations are not supported with ALTER TABLE:
+
+- ADD DEFAULT PARTITION
+- ADD PARTITION
+- DROP DEFAULT PARTITION
+- DROP PARTITION
+- SPLIT PARTITION
+- All operations that involve modifying subpartitions.
+
+## Limitations<a id="limitations"></a>
+
+HAWQ does not support using `ALTER TABLE` to `ADD` or `DROP` a column in an existing Parquet table.
+
+## Parameters<a id="altertable__section4"></a>
+
+ONLY
+Only perform the operation on the table name specified. If the `ONLY` keyword is not used, the operation will be performed on the named table and any child table partitions associated with that table.
+
+ *name*
+The name (possibly schema-qualified) of an existing table to alter. If `ONLY` is specified, only that table is altered. If `ONLY` is not specified, the table and all its descendant tables (if any) are updated.
+
+**Note:** Constraints can only be added to an entire table, not to a partition. Because of that restriction, the *name* parameter can only contain a table name, not a partition name.
+
+ *column*
+Name of a new or existing column. Note that HAWQ distribution key columns must be treated with special care. Altering or dropping these columns can change the distribution policy for the table.
+
+ *new\_column*
+New name for an existing column.
+
+ *new\_name*
+New name for the table.
+
+ *type*
+Data type of the new column, or new data type for an existing column. If changing the data type of a HAWQ distribution key column, you are only allowed to change it to a compatible type (for example, `text` to `varchar` is OK, but `text` to `int` is not).
+
+ *table\_constraint*
+New table constraint for the table. Note that foreign key constraints are currently not supported in HAWQ. Also a table is only allowed one unique constraint and the uniqueness must be within the HAWQ distribution key.
+
+ *constraint\_name*
+Name of an existing constraint to drop.
+
+CASCADE
+Automatically drop objects that depend on the dropped column or constraint (for example, views referencing the column).
+
+RESTRICT
+Refuse to drop the column or constraint if there are any dependent objects. This is the default behavior.
+
+ALL
+Disable or enable all triggers belonging to the table including constraint related triggers. This requires superuser privilege.
+
+USER
+Disable or enable all user-created triggers belonging to the table.
+
+DISTRIBUTED RANDOMLY | DISTRIBUTED BY (*column*)
+Specifies the distribution policy for a table. The default is RANDOM distribution. Changing a distribution policy will cause the table data to be physically redistributed on disk, which can be resource intensive. If you declare the same distribution policy or change from random to hash distribution, data will not be redistributed unless you declare `SET WITH (REORGANIZE=true)`.
+
+REORGANIZE=true|false
+Use `REORGANIZE=true` when the distribution policy has not changed or when you have changed from a random to a hash distribution, and you want to redistribute the data anyways.
+
+ *parent\_table*
+A parent table to associate or de-associate with this table.
+
+ *new\_owner*
+The role name of the new owner of the table.
+
+ *new\_tablespace*
+The name of the tablespace to which the table will be moved.
+
+ *new\_schema*
+The name of the schema to which the table will be moved.
+
+ *parent\_table\_name*
+When altering a partitioned table, the name of the top-level parent table.
+
+ALTER \[DEFAULT\] PARTITION
+If altering a partition deeper than the first level of partitions, the `ALTER PARTITION` clause is used to specify which subpartition in the hierarchy you want to alter.
+
+DROP \[DEFAULT\] PARTITION
+**Note:** Cannot be used with multi-level partitions.
+
+Drops the specified partition. If the partition has subpartitions, the subpartitions are automatically dropped as well.
+
+TRUNCATE \[DEFAULT\] PARTITION
+Truncates the specified partition. If the partition has subpartitions, the subpartitions are automatically truncated as well.
+
+RENAME \[DEFAULT\] PARTITION
+Changes the partition name of a partition (not the relation name). Partitioned tables are created using the naming convention: `<` *parentname* `>_<` *level* `>_prt_<` *partition\_name* `>`.
+
+ADD DEFAULT PARTITION
+**Note:** Cannot be used with multi-level partitions.
+
+Adds a default partition to an existing partition design. When data does not match to an existing partition, it is inserted into the default partition. Partition designs that do not have a default partition will reject incoming rows that do not match to an existing partition. Default partitions must be given a name.
+
+ADD PARTITION
+**Note:** Cannot be used with multi-level partitions.
+
+*partition\_element* - Using the existing partition type of the table (range or list), defines the boundaries of new partition you are adding.
+
+*name* - A name for this new partition.
+
+**VALUES** - For list partitions, defines the value(s) that the partition will contain.
+
+**START** - For range partitions, defines the starting range value for the partition. By default, start values are `INCLUSIVE`. For example, if you declared a start date of '`2008-01-01`', then the partition would contain all dates greater than or equal to '`2008-01-01`'. Typically the data type of the `START` expression is the same type as the partition key column. If that is not the case, then you must explicitly cast to the intended data type.
+
+**END** - For range partitions, defines the ending range value for the partition. By default, end values are `EXCLUSIVE`. For example, if you declared an end date of '`2008-02-01`', then the partition would contain all dates less than but not equal to '`2008-02-01`'. Typically the data type of the `END` expression is the same type as the partition key column. If that is not the case, then you must explicitly cast to the intended data type.
+
+**WITH** - Sets the table storage options for a partition. For example, you may want older partitions to be append-only tables and newer partitions to be regular heap tables. See `CREATE TABLE` for a description of the storage options.
+
+**TABLESPACE** - The name of the tablespace in which the partition is to be created.
+
+*subpartition\_spec* - Only allowed on partition designs that were created without a subpartition template. Declares a subpartition specification for the new partition you are adding. If the partitioned table was originally defined using a subpartition template, then the template will be used to generate the subpartitions automatically.
+
+EXCHANGE \[DEFAULT\] PARTITION
+Exchanges another table into the partition hierarchy into the place of an existing partition. In a multi-level partition design, you can only exchange the lowest level partitions (those that contain data).
+
+**WITH TABLE** *table\_name* - The name of the table you are swapping in to the partition design.
+
+**WITH** | **WITHOUT VALIDATION** - Validates that the data in the table matches the `CHECK` constraint of the partition you are exchanging. The default is to validate the data against the `CHECK` constraint.
+
+SET SUBPARTITION TEMPLATE
+Modifies the subpartition template for an existing partition. After a new subpartition template is set, all new partitions added will have the new subpartition design (existing partitions are not modified).
+
+SPLIT DEFAULT PARTITION
+**Note:** Cannot be used with multi-level partitions.
+
+Splits a default partition. In a multi-level partition design, you can only split the lowest level default partitions (those that contain data). Splitting a default partition creates a new partition containing the values specified and leaves the default partition containing any values that do not match to an existing partition.
+
+**AT** - For list partitioned tables, specifies a single list value that should be used as the criteria for the split.
+
+**START** - For range partitioned tables, specifies a starting value for the new partition.
+
+**END** - For range partitioned tables, specifies an ending value for the new partition.
+
+**INTO** - Allows you to specify a name for the new partition. When using the `INTO` clause to split a default partition, the second partition name specified should always be that of the existing default partition. If you do not know the name of the default partition, you can look it up using the `pg_partitions` view.
+
+SPLIT PARTITION
+**Note:** Cannot be used with multi-level partitions.
+
+Splits an existing partition into two partitions. In a multi-level partition design, you can only split the lowest level partitions (those that contain data).
+
+**AT** - Specifies a single value that should be used as the criteria for the split. The partition will be divided into two new partitions with the split value specified being the starting range for the *latter* partition.
+
+**INTO** - Allows you to specify names for the two new partitions created by the split.
+
+ *partition\_name*
+The given name of a partition.
+
+FOR (RANK(number))
+For range partitions, the rank of the partition in the range.
+
+FOR ('*value*')
+Specifies a partition by declaring a value that falls within the partition boundary specification. If the value declared with `FOR` matches to both a partition and one of its subpartitions (for example, if the value is a date and the table is partitioned by month and then by day), then `FOR` will operate on the first level where a match is found (for example, the monthly partition). If your intent is to operate on a subpartition, you must declare so as follows:
+
+``` pre
+ALTER TABLE name ALTER PARTITION FOR ('2008-10-01') DROP PARTITION FOR ('2008-10-01');
+```
+
+## Notes<a id="notes"></a>
+
+Take special care when altering or dropping columns that are part of the HAWQ distribution key as this can change the distribution policy for the table. HAWQ does not currently support foreign key constraints.
+
+**Note:** Note: The table name specified in the `ALTER TABLE` command cannot be the name of a partition within a table.
+
+Adding a `CHECK` or `NOT NULL` constraint requires scanning the table to verify that existing rows meet the constraint.
+
+When a column is added with `ADD COLUMN`, all existing rows in the table are initialized with the column’s default value (`NULL` if no `DEFAULT` clause is specified). Adding a column with a non-null default or changing the type of an existing column will require the entire table to be rewritten. This may take a significant amount of time for a large table; and it will temporarily require double the disk space.
+
+You can specify multiple changes in a single `ALTER TABLE` command, which will be done in a single pass over the table.
+
+The `DROP COLUMN` form does not physically remove the column, but simply makes it invisible to SQL operations. Subsequent insert and update operations in the table will store a null value for the column. Thus, dropping a column is quick but it will not immediately reduce the on-disk size of your table, as the space occupied by the dropped column is not reclaimed. The space will be reclaimed over time as existing rows are updated.
+
+The fact that `ALTER TYPE` requires rewriting the whole table is sometimes an advantage, because the rewriting process eliminates any dead space in the table. For example, to reclaim the space occupied by a dropped column immediately, the fastest way is: `ALTER TABLE table ALTER COLUMN anycol TYPE sametype;` Where *anycol* is any remaining table column and *sametype* is the same type that column already has. This results in no semantically-visible change in the table, but the command forces rewriting, which gets rid of no-longer-useful data.
+
+If a table is partitioned or has any descendant tables, it is not permitted to add, rename, or change the type of a column in the parent table without doing the same to the descendants. This ensures that the descendants always have columns matching the parent.
+
+A recursive `DROP COLUMN` operation will remove a descendant table’s column only if the descendant does not inherit that column from any other parents and never had an independent definition of the column. A nonrecursive `DROP COLUMN` (`ALTER TABLE ONLY ... DROP COLUMN`) never removes any descendant columns, but instead marks them as independently defined rather than inherited.
+
+The `OWNER` action never recurse to descendant tables; that is, they always act as though `ONLY` were specified. Adding a constraint can recurse only for `CHECK` constraints.
+
+Changing any part of a system catalog table is not permitted.
+
+## Examples<a id="examples"></a>
+
+Add a column to a table:
+
+``` pre
+ALTER TABLE distributors ADD COLUMN address varchar(30);
+```
+
+Rename an existing column:
+
+``` pre
+ALTER TABLE distributors RENAME COLUMN address TO city;
+```
+
+Rename an existing table:
+
+``` pre
+ALTER TABLE distributors RENAME TO suppliers;
+```
+
+Add a not-null constraint to a column:
+
+``` pre
+ALTER TABLE distributors ALTER COLUMN street SET NOT NULL;
+```
+
+Add a check constraint to a table:
+
+``` pre
+ALTER TABLE distributors ADD CONSTRAINT zipchk CHECK (char_length(zipcode) = 5);
+```
+
+Move a table to a different schema:
+
+``` pre
+ALTER TABLE myschema.distributors SET SCHEMA yourschema;
+```
+
+Add a new partition to a partitioned table:
+
+``` pre
+ALTER TABLE sales ADD PARTITION
+ START (date '2009-02-01') INCLUSIVE
+ END (date '2009-03-01') EXCLUSIVE;
+```
+
+Add a default partition to an existing partition design:
+
+``` pre
+ALTER TABLE sales ADD DEFAULT PARTITION other;
+```
+
+Rename a partition:
+
+``` pre
+ALTER TABLE sales RENAME PARTITION FOR ('2008-01-01') TO jan08;
+```
+
+Drop the first (oldest) partition in a range sequence:
+
+``` pre
+ALTER TABLE sales DROP PARTITION FOR (RANK(1));
+```
+
+Exchange a table into your partition design:
+
+``` pre
+ALTER TABLE sales EXCHANGE PARTITION FOR ('2008-01-01') WITH TABLE jan08;
+```
+
+Split the default partition (where the existing default partition’s name is `other`) to add a new monthly partition for January 2009:
+
+``` pre
+ALTER TABLE sales SPLIT DEFAULT PARTITION
+ START ('2009-01-01') INCLUSIVE
+ END ('2009-02-01') EXCLUSIVE
+ INTO (PARTITION jan09, PARTITION other);
+```
+
+Split a monthly partition into two with the first partition containing dates January 1-15 and the second partition containing dates January 16-31:
+
+``` pre
+ALTER TABLE sales SPLIT PARTITION FOR ('2008-01-01')
+ AT ('2008-01-16')
+ INTO (PARTITION jan081to15, PARTITION jan0816to31);
+```
+
+## Compatibility<a id="compat"></a>
+
+The `ADD`, `DROP`, and `SET DEFAULT` forms conform with the SQL standard. The other forms are HAWQ extensions of the SQL standard. Also, the ability to specify more than one manipulation in a single `ALTER TABLE` command is an extension. `ALTER TABLE DROP COLUMN` can be used to drop the only column of a table, leaving a zero-column table. This is an extension of SQL, which disallows zero-column tables.
+
+## See Also<a id="altertable__section8"></a>
+
+[CREATE TABLE](CREATE-TABLE.html), [DROP TABLE](DROP-TABLE.html)
diff --git a/reference/sql/ALTER-TABLESPACE.html.md.erb b/reference/sql/ALTER-TABLESPACE.html.md.erb
new file mode 100644
index 0000000..31d1c96
--- /dev/null
+++ b/reference/sql/ALTER-TABLESPACE.html.md.erb
@@ -0,0 +1,55 @@
+---
+title: ALTER TABLESPACE
+---
+
+Changes the definition of a tablespace.
+
+## Synopsis<a id="synopsis"></a>
+
+``` pre
+ALTER TABLESPACE name RENAME TO newname
+
+ALTER TABLESPACE name OWNER TO newowner
+
+```
+
+## Description<a id="desc"></a>
+
+`ALTER TABLESPACE` changes the definition of a tablespace.
+
+You must own the tablespace to use `ALTER TABLESPACE`. To alter the owner, you must also be a direct or indirect member of the new owning role. (Note that superusers have these privileges automatically.)
+
+## Parameters<a id="altertablespace__section4"></a>
+
+ *name*
+The name of an existing tablespace.
+
+ *newname*
+The new name of the tablespace. The new name cannot begin with *pg\_* (reserved for system tablespaces).
+
+ *newowner*
+The new owner of the tablespace.
+
+## Examples<a id="altertablespace__section5"></a>
+
+Rename tablespace `index_space` to `fast_raid`:
+
+``` pre
+ALTER TABLESPACE index_space RENAME TO fast_raid;
+```
+
+Change the owner of tablespace `index_space`:
+
+``` pre
+ALTER TABLESPACE index_space OWNER TO mary;
+```
+
+## Compatibility<a id="altertablespace__section6"></a>
+
+There is no `ALTER TABLESPACE` statement in the SQL standard.
+
+## See Also<a id="see"></a>
+
+[CREATE TABLESPACE](CREATE-TABLESPACE.html), [DROP TABLESPACE](DROP-TABLESPACE.html)
+
+
diff --git a/reference/sql/ALTER-TYPE.html.md.erb b/reference/sql/ALTER-TYPE.html.md.erb
new file mode 100644
index 0000000..2717a1e
--- /dev/null
+++ b/reference/sql/ALTER-TYPE.html.md.erb
@@ -0,0 +1,54 @@
+---
+title: ALTER TYPE
+---
+
+Changes the definition of a data type.
+
+## Synopsis<a id="synopsis"></a>
+
+``` pre
+ALTER TYPE name
+ OWNER TO new_owner | SET SCHEMA new_schema
+
+```
+
+## Description<a id="desc"></a>
+
+ `ALTER TYPE` changes the definition of an existing type. You can change the owner and the schema of a type.
+
+You must own the type to use `ALTER TYPE`. To change the schema of a type, you must also have `CREATE` privilege on the new schema. To alter the owner, you must also be a direct or indirect member of the new owning role, and that role must have `CREATE` privilege on the type's schema. (These restrictions enforce that altering the owner does not do anything that could be done by dropping and recreating the type. However, a superuser can alter ownership of any type.)
+
+## Parameters<a id="altertype__section4"></a>
+
+ *name*
+The name (optionally schema-qualified) of an existing type to alter.
+
+ *new\_owner*
+The user name of the new owner of the type.
+
+ *new\_schema*
+The new schema for the type.
+
+## Examples<a id="altertype__section5"></a>
+
+To change the owner of the user-defined type `email` to `joe`:
+
+``` pre
+ALTER TYPE email OWNER TO joe;
+```
+
+To change the schema of the user-defined type `email` to `customers`:
+
+``` pre
+ALTER TYPE email SET SCHEMA customers;
+```
+
+## Compatibility<a id="altertype__section6"></a>
+
+There is no `ALTER TYPE` statement in the SQL standard.
+
+## See Also<a id="see"></a>
+
+[CREATE TYPE](CREATE-TYPE.html), [DROP TYPE](DROP-TYPE.html)
+
+
diff --git a/reference/sql/ALTER-USER.html.md.erb b/reference/sql/ALTER-USER.html.md.erb
new file mode 100644
index 0000000..dadfcbf
--- /dev/null
+++ b/reference/sql/ALTER-USER.html.md.erb
@@ -0,0 +1,44 @@
+---
+title: ALTER USER
+---
+
+Changes the definition of a database role (user).
+
+## Synopsis<a id="alteruser__section2"></a>
+
+``` pre
+ALTER USER name RENAME TO newname
+
+ALTER USER name SET config_parameter {TO | =} {value | DEFAULT}
+
+ALTER USER name RESET config_parameter
+
+ALTER USER name [ [WITH] option [ ... ] ]
+```
+
+where *option* can be:
+
+``` pre
+ SUPERUSER | NOSUPERUSER
+ | CREATEDB | NOCREATEDB
+ | CREATEROLE | NOCREATEROLE
+ | CREATEUSER | NOCREATEUSER
+ | INHERIT | NOINHERIT
+ | LOGIN | NOLOGIN
+ | [ ENCRYPTED | UNENCRYPTED ] PASSWORD 'password'
+ | VALID UNTIL 'timestamp'
+```
+
+## Description<a id="alteruser__section3"></a>
+
+`ALTER USER` is a deprecated command but is still accepted for historical reasons. It is an alias for `ALTER ROLE`. See `ALTER ROLE` for more information.
+
+## Compatibility<a id="alteruser__section4"></a>
+
+The `ALTER USER` statement is a HAWQ extension. The SQL standard leaves the definition of users to the implementation.
+
+## See Also<a id="see"></a>
+
+[ALTER ROLE](ALTER-ROLE.html)
+
+
diff --git a/reference/sql/ANALYZE.html.md.erb b/reference/sql/ANALYZE.html.md.erb
new file mode 100644
index 0000000..a213221
--- /dev/null
+++ b/reference/sql/ANALYZE.html.md.erb
@@ -0,0 +1,75 @@
+---
+title: ANALYZE
+---
+
+Collects statistics about a database.
+
+## Synopsis<a id="synopsis"></a>
+
+``` pre
+ANALYZE [VERBOSE] [ROOTPARTITION] table [ (column [, ...] ) ]]
+```
+
+## Description<a id="desc"></a>
+
+`ANALYZE` collects statistics about the contents of tables in the database, and stores the results in the system table `pg_statistic`. Subsequently, the query planner uses these statistics to help determine the most efficient execution plans for queries.
+
+With no parameter, `ANALYZE` examines every table in the current database. With a parameter, `ANALYZE` examines only that table. It is further possible to give a list of column names, in which case only the statistics for those columns are collected.
+
+## Parameters<a id="params"></a>
+
+VERBOSE
+Enables display of progress messages. When specified, `ANALYZE` emits progress messages to indicate which table is currently being processed. Various statistics about the tables are printed as well.
+
+ROOTPARTITION
+For partitioned tables, `ANALYZE` on the parent (the root in multi-level partitioning) table without this option will collect statistics on each individual leaf partition as well as the global partition table, both of which are needed for query planning. In scenarios when all the individual child partitions have up-to-date statistics (for example, after loading and analyzing a daily partition), the `ROOTPARTITION` option can be used to collect only the global stats on the partition table. This could save the time of re-analyzing each individual leaf partition.
+
+If you use `ROOTPARTITION` on a non-root or non-partitioned table, `ANALYZE` will skip the option and issue a warning. You can also analyze all root partition tables in the database by using `ROOTPARTITION ALL`
+
+**Note:** Use `ROOTPARTITION ALL` to analyze all root partition tables in the database.
+
+ *table*
+The name (possibly schema-qualified) of a specific table to analyze. Defaults to all tables in the current database.
+
+ *column*
+The name of a specific column to analyze. Defaults to all columns.
+
+## Notes<a id="notes"></a>
+
+It is a good idea to run `ANALYZE` periodically, or just after making major changes in the contents of a table. Accurate statistics will help the query planner to choose the most appropriate query plan, and thereby improve the speed of query processing. A common strategy is to run `VACUUM` and `ANALYZE` once a day during a low-usage time of day.
+
+`ANALYZE` requires only a read lock on the target table, so it can run in parallel with other activity on the table.
+
+`ANALYZE` skips tables if the user is not the table owner or database owner.
+
+The statistics collected by `ANALYZE` usually include a list of some of the most common values in each column and a histogram showing the approximate data distribution in each column. One or both of these may be omitted if `ANALYZE` deems them uninteresting (for example, in a unique-key column, there are no common values) or if the column data type does not support the appropriate operators.
+
+For large tables, `ANALYZE` takes a random sample of the table contents, rather than examining every row. This allows even very large tables to be analyzed in a small amount of time. Note, however, that the statistics are only approximate, and will change slightly each time `ANALYZE` is run, even if the actual table contents did not change. This may result in small changes in the planner’s estimated costs shown by `EXPLAIN`. In rare situations, this non-determinism will cause the query optimizer to choose a different query plan between runs of `ANALYZE`. To avoid this, raise the amount of statistics collected by `ANALYZE` by adjusting the `default_statistics_target` configuration parameter, or on a column-by-column basis by setting the per-column statistics target with `ALTER TABLE ... ALTER COLUMN ... SET STATISTICS` (see `ALTER TABLE`). The target value sets the maximum number of entries in the most-common-value list and the maximum number of bins in the histogram. The default target value is 10, but this can be adjusted up or down to trade off accuracy of planner estimates against the time taken for `ANALYZE` and the amount of space occupied in `pg_statistic`. In particular, setting the statistics target to zero disables collection of statistics for that column. It may be useful to do that for columns that are never used as part of the `WHERE`, `GROUP BY`, or `ORDER BY` clauses of queries, since the planner will have no use for statistics on such columns.
+
+The largest statistics target among the columns being analyzed determines the number of table rows sampled to prepare the statistics. Increasing the target causes a proportional increase in the time and space needed to do `ANALYZE`.
+
+The `pxf_enable_stat_collection` server configuration parameter determines if `ANALYZE` calculates statistics for PXF readable tables. When `pxf_enable_stat_collection` is true, the default setting, `ANALYZE` estimates the number of tuples in the table from the total size of the table, the size of the first fragment, and the number of tuples in the first fragment. Then it builds a sample table and calculates statistics for the PXF table by running statistics queries on the sample table, the same as it does with native tables. A sample table is always created to calculate PXF table statistics, even when the table has a small number of rows.
+
+The `pxf_stat_max_fragments` configuration parameter, default 100, sets the maximum number of fragments that are sampled to build the sample table. Setting `pxf_stat_max_fragments` to a higher value provides a more uniform sample, but decreases `ANALYZE` performance. Setting it to a lower value increases performance, but the statistics are calculated on a less uniform sample.
+
+When `pxf_stat_max_fragments` is false, `ANALYZE` outputs a message to warn that it is skipping the PXF table because `pxf_stat_max_fragments` is turned off.
+
+There may be situations where the remote statistics retrieval could fail to perform a task on a PXF table. For example, if a PXF Java component is down, the remote statistics retrieval might not occur, and the database transaction would not succeed. In these cases, the statistics remain with the default external table values.
+
+## Examples<a id="examples"></a>
+
+Collect statistics for the table `mytable`:
+
+``` pre
+ANALYZE mytable;
+```
+
+## Compatibility<a id="compat"></a>
+
+There is no ANALYZE statement in the SQL standard.
+
+## See Also<a id="see"></a>
+
+[ALTER TABLE](ALTER-TABLE.html), [EXPLAIN](EXPLAIN.html), [VACUUM](VACUUM.html)
+
+
diff --git a/reference/sql/BEGIN.html.md.erb b/reference/sql/BEGIN.html.md.erb
new file mode 100644
index 0000000..16a5d2f
--- /dev/null
+++ b/reference/sql/BEGIN.html.md.erb
@@ -0,0 +1,58 @@
+---
+title: BEGIN
+---
+
+Starts a transaction block.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+BEGIN [WORK | TRANSACTION] [SERIALIZABLE | REPEATABLE READ | READ COMMITTED | READ UNCOMMITTED]
+ [READ WRITE | READ ONLY]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`BEGIN` initiates a transaction block, that is, all statements after a `BEGIN` command will be executed in a single transaction until an explicit `COMMIT` or `ROLLBACK` is given. By default (without `BEGIN`), HAWQ executes transactions in autocommit mode, that is, each statement is executed in its own transaction and a commit is implicitly performed at the end of the statement (if execution was successful, otherwise a rollback is done).
+
+Statements are executed more quickly in a transaction block, because transaction start/commit requires significant CPU and disk activity. Execution of multiple statements inside a transaction is also useful to ensure consistency when making several related changes: other sessions will be unable to see the intermediate states wherein not all the related updates have been done.
+
+## Parameters<a id="topic1__section4"></a>
+
+WORK
+TRANSACTION
+Optional key words. They have no effect.
+
+SERIALIZABLE
+REPEATABLE READ
+READ COMMITTED
+READ UNCOMMITTED
+The SQL standard defines four transaction isolation levels: `READ COMMITTED`, `READ UNCOMMITTED`, `SERIALIZABLE`, and `REPEATABLE READ`. The default behavior is that a statement can only see rows committed before it began (`READ COMMITTED`). In HAWQ, `READ UNCOMMITTED` is treated the same as `READ COMMITTED`. `SERIALIZABLE` is supported the same as `REPEATABLE READ` wherein all statements of the current transaction can only see rows committed before the first statement was executed in the transaction. `SERIALIZABLE` is the strictest transaction isolation. This level emulates serial transaction execution, as if transactions had been executed one after another, serially, rather than concurrently. Applications using this level must be prepared to retry transactions due to serialization failures.
+
+READ WRITE
+READ ONLY
+Determines whether the transaction is read/write or read-only. Read/write is the default. When a transaction is read-only, the following SQL commands are disallowed: `INSERT`, `UPDATE`, `DELETE`, and `COPY FROM` if the table they would write to is not a temporary table; all `CREATE`, `ALTER`, and `DROP` commands; `GRANT`, `REVOKE`, `TRUNCATE`; and `EXPLAIN ANALYZE` and `EXECUTE` if the command they would execute is among those listed.
+
+## Notes<a id="topic1__section5"></a>
+
+Use [COMMIT](COMMIT.html) or [ROLLBACK](ROLLBACK.html) to terminate a transaction block.
+
+Issuing `BEGIN` when already inside a transaction block will provoke a warning message. The state of the transaction is not affected. To nest transactions within a transaction block, use savepoints (see [SAVEPOINT](SAVEPOINT.html)).
+
+## Examples<a id="topic1__section6"></a>
+
+To begin a transaction block:
+
+``` pre
+BEGIN;
+```
+
+## Compatibility<a id="topic1__section7"></a>
+
+`BEGIN` is a HAWQ language extension. It is equivalent to the SQL-standard command `START TRANSACTION lang="EN"`.
+
+Incidentally, the `BEGIN` key word is used for a different purpose in embedded SQL. You are advised to be careful about the transaction semantics when porting database applications.
+
+## See Also<a id="topic1__section8"></a>
+
+[COMMIT](COMMIT.html), [ROLLBACK](ROLLBACK.html), [SAVEPOINT](SAVEPOINT.html)
diff --git a/reference/sql/CHECKPOINT.html.md.erb b/reference/sql/CHECKPOINT.html.md.erb
new file mode 100644
index 0000000..bbc2ee1
--- /dev/null
+++ b/reference/sql/CHECKPOINT.html.md.erb
@@ -0,0 +1,23 @@
+---
+title: CHECKPOINT
+---
+
+Forces a transaction log checkpoint.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+CHECKPOINT
+```
+
+## Description<a id="topic1__section3"></a>
+
+Write-Ahead Logging (WAL) puts a checkpoint in the transaction log every so often. The automatic checkpoint interval is set per HAWQ segment instance by the server configuration parameters *checkpoint\_segments* and *checkpoint\_timeout*. The `CHECKPOINT` command forces an immediate checkpoint when the command is issued, without waiting for a scheduled checkpoint.
+
+A checkpoint is a point in the transaction log sequence at which all data files have been updated to reflect the information in the log. All data files will be flushed to disk.
+
+Only superusers may call `CHECKPOINT`. The command is not intended for use during normal operation.
+
+## Compatibility<a id="topic1__section4"></a>
+
+The `CHECKPOINT` command is a HAWQ language extension.
diff --git a/reference/sql/CLOSE.html.md.erb b/reference/sql/CLOSE.html.md.erb
new file mode 100644
index 0000000..43d2997
--- /dev/null
+++ b/reference/sql/CLOSE.html.md.erb
@@ -0,0 +1,45 @@
+---
+title: CLOSE
+---
+
+Closes a cursor.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+CLOSE cursor_name
+
+```
+
+## Description<a id="topic1__section3"></a>
+
+`CLOSE` frees the resources associated with an open cursor. After the cursor is closed, no subsequent operations are allowed on it. A cursor should be closed when it is no longer needed.
+
+Every non-holdable open cursor is implicitly closed when a transaction is terminated by `COMMIT` or `ROLLBACK`. A holdable cursor is implicitly closed if the transaction that created it aborts via `ROLLBACK`. If the creating transaction successfully commits, the holdable cursor remains open until an explicit `CLOSE` is executed, or the client disconnects.
+
+## Parameters<a id="topic1__section4"></a>
+
+ *cursor\_name*
+The name of an open cursor to close.
+
+## Notes<a id="topic1__section5"></a>
+
+HAWQ does not have an explicit `OPEN` cursor statement. A cursor is considered open when it is declared. Use the `DECLARE` statement to declare (and open) a cursor.
+
+You can see all available cursors by querying the `pg_cursors` system view.
+
+## Examples<a id="topic1__section6"></a>
+
+Close the cursor `portala`:
+
+``` pre
+CLOSE portala;
+```
+
+## Compatibility<a id="topic1__section7"></a>
+
+`CLOSE` is fully conforming with the SQL standard.
+
+## See Also<a id="topic1__section8"></a>
+
+[DECLARE](DECLARE.html), [FETCH](FETCH.html)
diff --git a/reference/sql/COMMIT.html.md.erb b/reference/sql/COMMIT.html.md.erb
new file mode 100644
index 0000000..dee91d3
--- /dev/null
+++ b/reference/sql/COMMIT.html.md.erb
@@ -0,0 +1,43 @@
+---
+title: COMMIT
+---
+
+Commits the current transaction.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+COMMIT [WORK | TRANSACTION]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`COMMIT` commits the current transaction. All changes made by the transaction become visible to others and are guaranteed to be durable if a crash occurs.
+
+## Parameters<a id="topic1__section4"></a>
+
+WORK
+TRANSACTION
+Optional key words. They have no effect.
+
+## Notes<a id="topic1__section5"></a>
+
+Use [ROLLBACK](ROLLBACK.html) to abort a transaction.
+
+Issuing `COMMIT` when not inside a transaction does no harm, but it will provoke a warning message.
+
+## Examples<a id="topic1__section6"></a>
+
+To commit the current transaction and make all changes permanent:
+
+``` pre
+COMMIT;
+```
+
+## Compatibility<a id="topic1__section7"></a>
+
+The SQL standard only specifies the two forms `COMMIT` and `COMMIT WORK`. Otherwise, this command is fully conforming.
+
+## See Also<a id="topic1__section8"></a>
+
+[BEGIN](BEGIN.html), [END](END.html), [ROLLBACK](ROLLBACK.html)
diff --git a/reference/sql/COPY.html.md.erb b/reference/sql/COPY.html.md.erb
new file mode 100644
index 0000000..792cfe4
--- /dev/null
+++ b/reference/sql/COPY.html.md.erb
@@ -0,0 +1,253 @@
+---
+title: COPY
+---
+
+Copies data between a file and a table.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+COPY table [(column [, ...])] FROM {'file' | STDIN}
+ [ [WITH]
+ [OIDS]
+ [HEADER]
+ [DELIMITER [ AS ] 'delimiter']
+ [NULL [ AS ] 'null string']
+ [ESCAPE [ AS ] 'escape' | 'OFF']
+ [NEWLINE [ AS ] 'LF' | 'CR' | 'CRLF']
+ [CSV [QUOTE [ AS ] 'quote']
+ [FORCE NOT NULL column [, ...]]
+ [FILL MISSING FIELDS]
+ [[LOG ERRORS INTO error_table [KEEP]
+ SEGMENT REJECT LIMIT count [ROWS | PERCENT] ]
+
+COPY {table [(column [, ...])] | (query)} TO {'file' | STDOUT}
+ [ [WITH]
+ [OIDS]
+ [HEADER]
+ [DELIMITER [ AS ] 'delimiter']
+ [NULL [ AS ] 'null string']
+ [ESCAPE [ AS ] 'escape' | 'OFF']
+ [CSV [QUOTE [ AS ] 'quote']
+ [FORCE QUOTE column [, ...]] ]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`COPY` moves data between HAWQ tables and standard file-system files. `COPY TO` copies the contents of a table to a file, while `COPY FROM` copies data from a file to a table (appending the data to whatever is in the table already). `COPY TO` can also copy the results of a `SELECT` query.
+
+If a list of columns is specified, `COPY` will only copy the data in the specified columns to or from the file. If there are any columns in the table that are not in the column list, `COPY FROM` will insert the default values for those columns.
+
+`COPY` with a file name instructs the HAWQ master host to directly read from or write to a file. The file must be accessible to the master host and the name must be specified from the viewpoint of the master host. When `STDIN` or `STDOUT` is specified, data is transmitted via the connection between the client and the master.
+
+If `SEGMENT REJECT LIMIT` is used, then a `COPY FROM` operation will operate in single row error isolation mode. In this release, single row error isolation mode only applies to rows in the input file with format errors — for example, extra or missing attributes, attributes of a wrong data type, or invalid client encoding sequences. Constraint errors such as violation of a `NOT NULL`, `CHECK`, or `UNIQUE` constraint will still be handled in 'all-or-nothing' input mode. The user can specify the number of error rows acceptable (on a per-segment basis), after which the entire `COPY FROM` operation will be aborted and no rows will be loaded. Note that the count of error rows is per-segment, not per entire load operation. If the per-segment reject limit is not reached, then all rows not containing an error will be loaded. If the limit is not reached, all good rows will be loaded and any error rows discarded. If you would like to keep error rows for further examination, you can optionally declare an error table using the `LOG ERRORS INTO` clause. Any rows containing a format error would then be logged to the specified error table.
+
+**Outputs**
+
+On successful completion, a `COPY` command returns a command tag of the form, where *count* is the number of rows copied:
+
+``` pre
+COPY count
+
+```
+
+If running a `COPY FROM` command in single row error isolation mode, the following notice message will be returned if any rows were not loaded due to format errors, where *count* is the number of rows rejected:
+
+``` pre
+NOTICE: Rejected count badly formatted rows.
+```
+
+## Parameters<a id="topic1__section5"></a>
+
+ *table*
+The name (optionally schema-qualified) of an existing table.
+
+ *column*
+An optional list of columns to be copied. If no column list is specified, all columns of the table will be copied.
+
+ *query*
+A `SELECT` or `VALUES` command whose results are to be copied. Note that parentheses are required around the query.
+
+ *file*
+The absolute path name of the input or output file.
+
+STDIN
+Specifies that input comes from the client application.
+
+STDOUT
+Specifies that output goes to the client application.
+
+OIDS
+Specifies copying the OID for each row. (An error is raised if OIDS is specified for a table that does not have OIDs, or in the case of copying a query.)
+
+ *delimiter*
+The single ASCII character that separates columns within each row (line) of the file. The default is a tab character in text mode, a comma in `CSV` mode.
+
+ *null string*
+The string that represents a null value. The default is `\N` (backslash-N) in text mode, and a empty value with no quotes in `CSV` mode. You might prefer an empty string even in text mode for cases where you don't want to distinguish nulls from empty strings. When using `COPY FROM`, any data item that matches this string will be stored as a null value, so you should make sure that you use the same string as you used with `COPY TO`.
+
+ *escape*
+Specifies the single character that is used for C escape sequences (such as `\n`,`\t`,`\100`, and so on) and for quoting data characters that might otherwise be taken as row or column delimiters. Make sure to choose an escape character that is not used anywhere in your actual column data. The default escape character is `\` (backslash) for text files or `"` (double quote) for CSV files, however it is possible to specify any other character to represent an escape. It is also possible to disable escaping on text-formatted files by specifying the value '`OFF'` as the escape value. This is very useful for data such as web log data that has many embedded backslashes that are not intended to be escapes.
+
+NEWLINE
+Specifies the newline used in your data files — `LF` (Line feed, 0x0A), `CR` (Carriage return, 0x0D), or `CRLF` (Carriage return plus line feed, 0x0D 0x0A). If not specified, a HAWQ segment will detect the newline type by looking at the first row of data it receives and using the first newline type encountered.
+
+CSV
+Selects Comma Separated Value (CSV) mode.
+
+HEADER
+Specifies that a file contains a header line with the names of each column in the file. On output, the first line contains the column names from the table, and on input, the first line is ignored.
+
+ *quote*
+Specifies the quotation character in CSV mode. The default is double-quote.
+
+FORCE QUOTE
+In `CSV COPY TO` mode, forces quoting to be used for all non-`NULL` values in each specified column. `NULL` output is never quoted.
+
+FORCE NOT NULL
+In `CSV COPY FROM` mode, process each specified column as though it were quoted and hence not a `NULL` value. For the default null string in `CSV` mode (nothing between two delimiters), this causes missing values to be evaluated as zero-length strings.
+
+FILL MISSING FIELDS
+In `COPY FROM` more for both `TEXT` and `CSV`, specifying `FILL MISSING FIELDS` will set missing trailing field values to `NULL` (instead of reporting an error) when a row of data has missing data fields at the end of a line or row. Blank rows, fields with a `NOT NULL` constraint, and trailing delimiters on a line will still report an error.
+
+LOG ERRORS INTO *error\_table* \[KEEP\]
+This is an optional clause that can precede a `SEGMENT REJECT LIMIT` clause to log information about rows with formatting errors. The `INTO error_table ` clause specifies an error table where rows with formatting errors will be logged when running in single row error isolation mode. You can then examine this error table to see error rows that were not loaded (if any). If the *error\_table* specified already exists, it will be used. If it does not exist, it will be automatically generated. If the command auto-generates the error table and no errors are produced, the default is to drop the error table after the operation completes unless `KEEP` is specified. If the table is auto-generated and the error limit is exceeded, the entire transaction is rolled back and no error data is saved. If you want the error table to persist in this case, create the error table prior to running the `COPY`. An error table is defined as follows:
+
+``` pre
+CREATE TABLE error_table_name ( cmdtime timestamptz, relname text,
+ filename text, linenum int, bytenum int, errmsg text,
+ rawdata text, rawbytes bytea ) DISTRIBUTED RANDOMLY;
+```
+
+SEGMENT REJECT LIMIT count \[ROWS | PERCENT\]
+Runs a `COPY FROM` operation in single row error isolation mode. If the input rows have format errors they will be discarded provided that the reject limit count is not reached on any HAWQ segment instance during the load operation. The reject limit count can be specified as number of rows (the default) or percentage of total rows (1-100). If `PERCENT` is used, each segment starts calculating the bad row percentage only after the number of rows specified by the parameter `gp_reject_percent_threshold` has been processed. The default for `gp_reject_percent_threshold` is 300 rows. Constraint errors such as violation of a `NOT NULL` or `CHECK` constraint will still be handled in 'all-or-nothing' input mode. If the limit is not reached, all good rows will be loaded and any error rows discarded.
+
+## Notes<a id="topic1__section6"></a>
+
+`COPY` can only be used with tables, not with views. However, you can write `COPY (SELECT * FROM viewname) TO ...`
+
+The `BINARY` key word causes all data to be stored/read as binary format rather than as text. It is somewhat faster than the normal text mode, but a binary-format file is less portable across machine architectures and HAWQ versions. Also, you cannot run `COPY FROM` in single row error isolation mode if the data is in binary format.
+
+You must have `SELECT` privilege on the table whose values are read by `COPY TO`, and insert privilege on the table into which values are inserted by `COPY FROM`.
+
+Files named in a `COPY` command are read or written directly by the database server, not by the client application. Therefore, they must reside on or be accessible to the HAWQ master host machine, not the client. They must be accessible to and readable or writable by the HAWQ system user (the user ID the server runs as), not the client. `COPY` naming a file is only allowed to database superusers, since it allows reading or writing any file that the server has privileges to access.
+
+`COPY FROM` will invoke any check constraints on the destination table. However, it will not invoke rewrite rules. Note that in this release, violations of constraints are not evaluated for single row error isolation mode.
+
+`COPY` input and output is affected by `DateStyle`. To ensure portability to other HAWQ installations that might use non-default `DateStyle` settings, `DateStyle` should be set to ISO before using `COPY TO`.
+
+By default, `COPY` stops operation at the first error. This should not lead to problems in the event of a `COPY TO`, but the target table will already have received earlier rows in a `COPY FROM`. These rows will not be visible or accessible, but they still occupy disk space. This may amount to a considerable amount of wasted disk space if the failure happened well into a large `COPY FROM` operation. You may wish to invoke `VACUUM` to recover the wasted space. Another option would be to use single row error isolation mode to filter out error rows while still loading good rows.
+
+COPY supports creating readable foreign tables with error tables. The default for concurrently inserting into the error table is 127. You can use error tables with foreign tables under the following circumstances:
+
+- Multiple foreign tables can use different error tables
+- Multiple foreign tables cannot use the same error table
+
+## File Formats<a id="topic1__section7"></a>
+
+File formats supported by `COPY`.
+
+**Text Format**
+When `COPY` is used without the `BINARY` or `CSV` options, the data read or written is a text file with one line per table row. Columns in a row are separated by the *delimiter* character (tab by default). The column values themselves are strings generated by the output function, or acceptable to the input function, of each attribute's data type. The specified null string is used in place of columns that are null. `COPY FROM` will raise an error if any line of the input file contains more or fewer columns than are expected. If `OIDS` is specified, the OID is read or written as the first column, preceding the user data columns.
+
+The data file has two reserved characters that have special meaning to `COPY`:
+
+- The designated delimiter character (tab by default), which is used to separate fields in the data file.
+- A UNIX-style line feed (`\n` or `0x0a`), which is used to designate a new row in the data file. It is strongly recommended that applications generating `COPY` data convert data line feeds to UNIX-style line feeds rather than Microsoft Windows style carriage return line feeds (`\r\n` or `0x0a 0x0d`).
+
+If your data contains either of these characters, you must escape the character so `COPY` treats it as data and not as a field separator or new row.
+
+By default, the escape character is a `\` (backslash) for text-formatted files and a `"` (double quote) for csv-formatted files. If you want to use a different escape character, you can do so using the `ESCAPE AS `clause. Make sure to choose an escape character that is not used anywhere in your data file as an actual data value. You can also disable escaping in text-formatted files by using `ESCAPE 'OFF'`.
+
+For example, suppose you have a table with three columns and you want to load the following three fields using COPY.
+
+- percentage sign = %
+- vertical bar = |
+- backslash = \\
+
+Your designated *delimiter* character is `|` (pipe character), and your designated *escape* character is `*` (asterisk). The formatted row in your data file would look like this:
+
+``` pre
+percentage sign = % | vertical bar = *| | backslash = \
+```
+
+Notice how the pipe character that is part of the data has been escaped using the asterisk character (\*). Also notice that we do not need to escape the backslash since we are using an alternative escape character.
+
+The following characters must be preceded by the escape character if they appear as part of a column value: the escape character itself, newline, carriage return, and the current delimiter character. You can specify a different escape character using the `ESCAPE AS` clause.
+
+**CSV Format**
+
+This format is used for importing and exporting the Comma Separated Value (CSV) file format used by many other programs, such as spreadsheets. Instead of the escaping used by HAWQ standard text mode, it produces and recognizes the common CSV escaping mechanism.
+
+The values in each record are separated by the `DELIMITER` character. If the value contains the delimiter character, the `QUOTE` character, the `ESCAPE` character (which is double quote by default), the `NULL` string, a carriage return, or line feed character, then the whole value is prefixed and suffixed by the `QUOTE` character. You can also use `FORCE QUOTE` to force quotes when outputting non-`NULL` values in specific columns.
+
+The CSV format has no standard way to distinguish a `NULL` value from an empty string. HAWQ `COPY` handles this by quoting. A `NULL` is output as the `NULL` string and is not quoted, while a data value matching the `NULL` string is quoted. Therefore, using the default settings, a `NULL` is written as an unquoted empty string, while an empty string is written with double quotes (""). Reading values follows similar rules. You can use `FORCE NOT NULL` to prevent `NULL` input comparisons for specific columns.
+
+Because backslash is not a special character in the `CSV` format, `\.`, the end-of-data marker, could also appear as a data value. To avoid any misinterpretation, a `\.` data value appearing as a lone entry on a line is automatically quoted on output, and on input, if quoted, is not interpreted as the end-of-data marker. If you are loading a file created by another application that has a single unquoted column and might have a value of `\.`, you might need to quote that value in the input file.
+
+**Note:** In `CSV` mode, all characters are significant. A quoted value surrounded by white space, or any characters other than `DELIMITER`, will include those characters. This can cause errors if you import data from a system that pads CSV lines with white space out to some fixed width. If such a situation arises you might need to preprocess the CSV file to remove the trailing white space, before importing the data into HAWQ.
+
+**Note:** `CSV` mode will both recognize and produce CSV files with quoted values containing embedded carriage returns and line feeds. Thus the files are not strictly one line per table row like text-mode files.
+
+**Note:** Many programs produce strange and occasionally perverse CSV files, so the file format is more a convention than a standard. Thus you might encounter some files that cannot be imported using this mechanism, and `COPY` might produce files that other programs cannot process.
+
+**Binary Format**
+
+The `BINARY` format consists of a file header, zero or more tuples containing the row data, and a file trailer. Headers and data are in network byte order.
+
+- **File Header** — The file header consists of 15 bytes of fixed fields, followed by a variable-length header extension area. The fixed fields are:
+ - **Signature** — 11-byte sequence PGCOPY\\n\\377\\r\\n\\0 — note that the zero byte is a required part of the signature. (The signature is designed to allow easy identification of files that have been munged by a non-8-bit-clean transfer. This signature will be changed by end-of-line-translation filters, dropped zero bytes, dropped high bits, or parity changes.)
+ - **Flags field** — 32-bit integer bit mask to denote important aspects of the file format. Bits are numbered from 0 (LSB) to 31 (MSB). Note that this field is stored in network byte order (most significant byte first), as are all the integer fields used in the file format. Bits 16-31 are reserved to denote critical file format issues; a reader should abort if it finds an unexpected bit set in this range. Bits 0-15 are reserved to signal backwards-compatible format issues; a reader should simply ignore any unexpected bits set in this range. Currently only one flag is defined, and the rest must be zero (Bit 16: 1 if data has OIDs, 0 if not).
+ - **Header extension area length** — 32-bit integer, length in bytes of remainder of header, not including self. Currently, this is zero, and the first tuple follows immediately. Future changes to the format might allow additional data to be present in the header. A reader should silently skip over any header extension data it does not know what to do with. The header extension area is envisioned to contain a sequence of self-identifying chunks. The flags field is not intended to tell readers what is in the extension area. Specific design of header extension contents is left for a later release.
+- **Tuples** — Each tuple begins with a 16-bit integer count of the number of fields in the tuple. (Presently, all tuples in a table will have the same count, but that might not always be true.) Then, repeated for each field in the tuple, there is a 32-bit length word followed by that many bytes of field data. (The length word does not include itself, and can be zero.) As a special case, -1 indicates a NULL field value. No value bytes follow in the NULL case.
+
+ There is no alignment padding or any other extra data between fields.
+
+ Presently, all data values in a COPY BINARY file are assumed to be in binary format (format code one). It is anticipated that a future extension may add a header field that allows per-column format codes to be specified.
+
+ If OIDs are included in the file, the OID field immediately follows the field-count word. It is a normal field except that it's not included in the field-count. In particular it has a length word — this will allow handling of 4-byte vs. 8-byte OIDs without too much pain, and will allow OIDs to be shown as null if that ever proves desirable.
+
+- **File Trailer** — The file trailer consists of a 16-bit integer word containing `-1`. This is easily distinguished from a tuple's field-count word. A reader should report an error if a field-count word is neither `-1` nor the expected number of columns. This provides an extra check against somehow getting out of sync with the data.
+
+## Examples<a id="topic1__section11"></a>
+
+Copy a table to the client using the vertical bar (|) as the field delimiter:
+
+``` pre
+COPY country TO STDOUT WITH DELIMITER '|';
+```
+
+Copy data from a file into the `country` table:
+
+``` pre
+COPY country FROM '/home/usr1/sql/country_data';
+```
+
+Copy into a file just the countries whose names start with 'A':
+
+``` pre
+COPY (SELECT * FROM country WHERE country_name LIKE 'A%') TO
+'/home/usr1/sql/a_list_countries.copy';
+```
+
+Create an error table called `err_sales` to use with single row error isolation mode:
+
+``` pre
+CREATE TABLE err_sales ( cmdtime timestamptz, relname text,
+filename text, linenum int, bytenum int, errmsg text, rawdata text, rawbytes bytea ) DISTRIBUTED RANDOMLY;
+```
+
+Copy data from a file into the `sales` table using single row error isolation mode:
+
+``` pre
+COPY sales FROM '/home/usr1/sql/sales_data' LOG ERRORS INTO
+err_sales SEGMENT REJECT LIMIT 10 ROWS;
+```
+
+## Compatibility<a id="topic1__section12"></a>
+
+There is no `COPY` statement in the SQL standard.
+
+## See Also<a id="topic1__section13"></a>
+
+[CREATE EXTERNAL TABLE](CREATE-EXTERNAL-TABLE.html)
diff --git a/reference/sql/CREATE-AGGREGATE.html.md.erb b/reference/sql/CREATE-AGGREGATE.html.md.erb
new file mode 100644
index 0000000..cdf4bc8
--- /dev/null
+++ b/reference/sql/CREATE-AGGREGATE.html.md.erb
@@ -0,0 +1,161 @@
+---
+title: CREATE AGGREGATE
+---
+
+Defines a new aggregate function.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+CREATE [ORDERED] AGGREGATE name (input_data_type [ , ... ])
+ ( SFUNC = sfunc,
+ STYPE = state_data_type
+ [, PREFUNC = prefunc]
+ [, FINALFUNC = ffunc]
+ [, INITCOND = initial_condition]
+ [, SORTOP = sort_operator] )
+```
+
+## Description<a id="topic1__section3"></a>
+
+`CREATE AGGREGATE` defines a new aggregate function. Some basic and commonly-used aggregate functions such as `count`, `min`, `max`, `sum`, `avg` and so on are already provided in HAWQ. If one defines new types or needs an aggregate function not already provided, then `CREATE AGGREGATE` can be used to provide the desired features.
+
+An aggregate function is identified by its name and input data types. Two aggregate functions in the same schema can have the same name if they operate on different input types. The name and input data types of an aggregate function must also be distinct from the name and input data types of every ordinary function in the same schema.
+
+An aggregate function is made from one, two or three ordinary functions (all of which must be `IMMUTABLE` functions):
+
+- A state transition function *sfunc*
+- An optional preliminary segment-level calculation function *prefunc*
+- An optional final calculation function *ffunc*
+
+These functions are used as follows:
+
+``` pre
+sfunc( internal-state, next-data-values ) ---> next-internal-state
+prefunc( internal-state, internal-state ) ---> next-internal-state
+ffunc( internal-state ) ---> aggregate-value
+```
+
+You can specify `PREFUNC` as method for optimizing aggregate execution. By specifying `PREFUNC`, the aggregate can be executed in parallel on segments first and then on the master. When a two-level execution is performed, `SFUNC` is executed on the segments to generate partial aggregate results, and `PREFUNC` is executed on the master to aggregate the partial results from segments. If single-level aggregation is performed, all the rows are sent to the master and `sfunc` is applied to the rows.
+
+Single-level aggregation and two-level aggregation are equivalent execution strategies. Either type of aggregation can be implemented in a query plan. When you implement the functions `prefunc` and `sfunc`, you must ensure that the invocation of `sfunc` on the segment instances followed by `prefunc` on the master produce the same result as single-level aggregation that sends all the rows to the master and then applies only the `sfunc` to the rows.
+
+HAWQ creates a temporary variable of data type *stype* to hold the current internal state of the aggregate function. At each input row, the aggregate argument values are calculated and the state transition function is invoked with the current state value and the new argument values to calculate a new internal state value. After all the rows have been processed, the final function is invoked once to calculate the aggregate return value. If there is no final function then the ending state value is returned as-is.
+
+An aggregate function can provide an optional initial condition, an initial value for the internal state value. This is specified and stored in the database as a value of type text, but it must be a valid external representation of a constant of the state value data type. If it is not supplied then the state value starts out `NULL`.
+
+If the state transition function is declared `STRICT`, then it cannot be called with `NULL` inputs. With such a transition function, aggregate execution behaves as follows. Rows with any null input values are ignored (the function is not called and the previous state value is retained). If the initial state value is `NULL`, then at the first row with all non-null input values, the first argument value replaces the state value, and the transition function is invoked at subsequent rows with all non-null input values. This is useful for implementing aggregates like `max`. Note that this behavior is only available when *state\_data\_type* is the same as the first *input\_data\_type*. When these types are different, you must supply a non-null initial condition or use a nonstrict transition function.
+
+If the state transition function is not declared `STRICT`, then it will be called unconditionally at each input row, and must deal with `NULL` inputs and `NULL` transition values for itself. This allows the aggregate author to have full control over the aggregate handling of `NULL` values.
+
+If the final function is declared `STRICT`, then it will not be called when the ending state value is `NULL`; instead a `NULL` result will be returned automatically. (This is the normal behavior of `STRICT` functions.) In any case the final function has the option of returning a `NULL` value. For example, the final function for `avg` returns `NULL` when it sees there were zero input rows.
+
+Single argument aggregate functions, such as min or max, can sometimes be optimized by looking into an index instead of scanning every input row. If this aggregate can be so optimized, indicate it by specifying a sort operator. The basic requirement is that the aggregate must yield the first element in the sort ordering induced by the operator; in other words:
+
+``` pre
+SELECT agg(col) FROM tab;
+```
+
+must be equivalent to:
+
+``` pre
+SELECT col FROM tab ORDER BY col USING sortop LIMIT 1;
+```
+
+Further assumptions are that the aggregate function ignores `NULL` inputs, and that it delivers a `NULL` result if and only if there were no non-null inputs. Ordinarily, a data type's `<` operator is the proper sort operator for `MIN`, and `>` is the proper sort operator for `MAX`. Note that the optimization will never actually take effect unless the specified operator is the "less than" or "greater than" strategy member of a B-tree index operator class.
+
+**Ordered Aggregates**
+
+If the optional qualification `ORDERED` appears, the created aggregate function is an *ordered aggregate*. In this case, the preliminary aggregation function, `prefunc` cannot be specified.
+
+An ordered aggregate is called with the following syntax.
+
+``` pre
+name ( arg [ , ... ] [ORDER BY sortspec [ , ...]] )
+```
+
+If the optional `ORDER BY` is omitted, a system-defined ordering is used. The transition function `sfunc` of an ordered aggregate function is called on its input arguments in the specified order and on a single segment. There is a new column `aggordered` in the `pg_aggregate` table to indicate the aggregate function is defined as an ordered aggregate.
+
+## Parameters<a id="topic1__section5"></a>
+
+ *name*
+The name (optionally schema-qualified) of the aggregate function to create.
+
+ *input\_data\_type*
+An input data type on which this aggregate function operates. To create a zero-argument aggregate function, write \* in place of the list of input data types. An example of such an aggregate is count(\*).
+
+ *sfunc*
+The name of the state transition function to be called for each input row. For an N-argument aggregate function, the *sfunc* must take N+1 arguments, the first being of type *state\_data\_type* and the rest matching the declared input data types of the aggregate. The function must return a value of type *state\_data\_type*. This function takes the current state value and the current input data values, and returns the next state value.
+
+ *state\_data\_type*
+The data type for the aggregate state value.
+
+ *prefunc*
+The name of a preliminary aggregation function. This is a function of two arguments, both of type *state\_data\_type*. It must return a value of *state\_data\_type*. A preliminary function takes two transition state values and returns a new transition state value representing the combined aggregation. In HAWQ, if the result of the aggregate function is computed in a segmented fashion, the preliminary aggregation function is invoked on the individual internal states in order to combine them into an ending internal state.
+
+Note that this function is also called in hash aggregate mode within a segment. Therefore, if you call this aggregate function without a preliminary function, hash aggregate is never chosen. Since hash aggregate is efficient, consider defining preliminary function whenever possible.
+
+PREFUNC is optional. If defined, it is executed on master. Input to PREFUNC is partial results from segments, and not the tuples. If PREFUNC is not defined, the aggregate cannot be executed in parallel. PREFUNC and gp\_enable\_multiphase\_agg are used as follows:
+
+- gp\_enable\_multiphase\_agg = off: SFUNC is executed sequentially on master. PREFUNC, even if defined, is unused.
+- gp\_enable\_multiphase\_agg = on and PREFUNC is defined: SFUNC is executed in parallel, on segments. PREFUNC is invoked on master to aggregate partial results from segments.
+
+ ``` pre
+ CREATE OR REPLACE FUNCTION my_avg_accum(bytea,bigint) returns bytea as 'int8_avg_accum' language internal strict immutable;
+ CREATE OR REPLACE FUNCTION my_avg_merge(bytea,bytea) returns bytea as 'int8_avg_amalg' language internal strict immutable;
+ CREATE OR REPLACE FUNCTION my_avg_final(bytea) returns numeric as 'int8_avg' language internal strict immutable;
+ CREATE AGGREGATE my_avg(bigint) ( stype = bytea,sfunc = my_avg_accum,prefunc = my_avg_merge,finalfunc = my_avg_final,initcond = '' );
+ ```
+
+ *ffunc*
+The name of the final function called to compute the aggregate result after all input rows have been traversed. The function must take a single argument of type `state_data_type`. The return data type of the aggregate is defined as the return type of this function. If `ffunc` is not specified, then the ending state value is used as the aggregate result, and the return type is *state\_data\_type*.
+
+ *initial\_condition*
+The initial setting for the state value. This must be a string constant in the form accepted for the data type *state\_data\_type*. If not specified, the state value starts out `NULL`.
+
+ *sort\_operator*
+The associated sort operator for a MIN- or MAX-like aggregate function. This is just an operator name (possibly schema-qualified). The operator is assumed to have the same input data types as the aggregate function (which must be a single-argument aggregate function).
+
+## Notes<a id="topic1__section6"></a>
+
+The ordinary functions used to define a new aggregate function must be defined first. Note that in this release of HAWQ, it is required that the *sfunc*, *ffunc*, and *prefunc* functions used to create the aggregate are defined as `IMMUTABLE`.
+
+Any compiled code (shared library files) for custom functions must be placed in the same location on every host in your HAWQ array (master and all segments). This location must also be in the `LD_LIBRARY_PATH` so that the server can locate the files.
+
+## Examples
+
+Create a sum of cubes aggregate:
+
+``` pre
+CREATE FUNCTION scube_accum(numeric, numeric) RETURNS numeric
+ AS 'select $1 + $2 * $2 * $2'
+ LANGUAGE SQL
+ IMMUTABLE
+ RETURNS NULL ON NULL INPUT;
+CREATE AGGREGATE scube(numeric) (
+ SFUNC = scube_accum,
+ STYPE = numeric,
+ INITCOND = 0 );
+```
+
+To test this aggregate:
+
+``` pre
+CREATE TABLE x(a INT);
+INSERT INTO x VALUES (1),(2),(3);
+SELECT scube(a) FROM x;
+```
+
+Correct answer for reference:
+
+``` pre
+SELECT sum(a*a*a) FROM x;
+```
+
+## Compatibility<a id="topic1__section8"></a>
+
+`CREATE AGGREGATE` is a HAWQ language extension. The SQL standard does not provide for user-defined aggregate functions.
+
+## See Also<a id="topic1__section9"></a>
+
+`ALTER AGGREGATE`, [DROP AGGREGATE](DROP-AGGREGATE.html), [CREATE FUNCTION](CREATE-FUNCTION.html)
diff --git a/reference/sql/CREATE-DATABASE.html.md.erb b/reference/sql/CREATE-DATABASE.html.md.erb
new file mode 100644
index 0000000..9debc83
--- /dev/null
+++ b/reference/sql/CREATE-DATABASE.html.md.erb
@@ -0,0 +1,81 @@
+---
+title: CREATE DATABASE
+---
+
+Creates a new database.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+CREATE DATABASE name [ [WITH] [OWNER [=] dbowner]
+ [TEMPLATE [=] template]
+ [ENCODING [=] encoding]
+ [TABLESPACE [=] tablespace]
+ [CONNECTION LIMIT [=] connlimit ] ]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`CREATE DATABASE` creates a new database. To create a database, you must be a superuser or have the special `CREATEDB` privilege.
+
+The creator becomes the owner of the new database by default. Superusers can create databases owned by other users by using the `OWNER` clause. They can even create databases owned by users with no special privileges. Non-superusers with `CREATEDB` privilege can only create databases owned by themselves.
+
+By default, the new database will be created by cloning the standard system database `template1`. A different template can be specified by writing `TEMPLATE name `. In particular, by writing `TEMPLATE template0`, you can create a clean database containing only the standard objects predefined by HAWQ. This is useful if you wish to avoid copying any installation-local objects that may have been added to `template1`.
+
+## Parameters<a id="topic1__section4"></a>
+
+ *name*
+The name of a database to create.
+
+**Note:** HAWQ reserves the database name "hcatalog" for system use.
+
+ *dbowner*
+The name of the database user who will own the new database, or `DEFAULT` to use the default owner (the user executing the command).
+
+ *template*
+The name of the template from which to create the new database, or `DEFAULT` to use the default template (*template1*).
+
+ *encoding*
+Character set encoding to use in the new database. Specify a string constant (such as `'SQL_ASCII'`), an integer encoding number, or `DEFAULT` to use the default encoding.
+
+ *tablespace*
+The name of the tablespace that will be associated with the new database, or `DEFAULT` to use the template database's tablespace. This tablespace will be the default tablespace used for objects created in this database.
+
+ *connlimit*
+The maximum number of concurrent connections posible. The default of -1 means there is no limitation.
+
+## Notes<a id="topic1__section5"></a>
+
+`CREATE DATABASE` cannot be executed inside a transaction block.
+
+When you copy a database by specifying its name as the template, no other sessions can be connected to the template database while it is being copied. New connections to the template database are locked out until `CREATE DATABASE` completes.
+
+The `CONNECTION LIMIT` is not enforced against superusers.
+
+## Examples<a id="topic1__section6"></a>
+
+To create a new database:
+
+``` pre
+CREATE DATABASE gpdb;
+```
+
+To create a database `sales` owned by user `salesapp` with a default tablespace of `salesspace`:
+
+``` pre
+CREATE DATABASE sales OWNER salesapp TABLESPACE salesspace;
+```
+
+To create a database `music` which supports the ISO-8859-1 character set:
+
+``` pre
+CREATE DATABASE music ENCODING 'LATIN1';
+```
+
+## Compatibility<a id="topic1__section7"></a>
+
+There is no `CREATE DATABASE` statement in the SQL standard. Databases are equivalent to catalogs, whose creation is implementation-defined.
+
+## See Also<a id="topic1__section8"></a>
+
+[DROP DATABASE](DROP-DATABASE.html)
diff --git a/reference/sql/CREATE-EXTERNAL-TABLE.html.md.erb b/reference/sql/CREATE-EXTERNAL-TABLE.html.md.erb
new file mode 100644
index 0000000..68d2f36
--- /dev/null
+++ b/reference/sql/CREATE-EXTERNAL-TABLE.html.md.erb
@@ -0,0 +1,334 @@
+---
+title: CREATE EXTERNAL TABLE
+---
+
+Defines a new external table.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+CREATE [READABLE] EXTERNAL TABLE table_name
+ ( column_name
+ data_type [, ...] | LIKE other_table )
+ LOCATION ('gpfdist://filehost[:port]/file_pattern[#transform]' [, ...])
+ | ('gpfdists://filehost[:port]/file_pattern[#transform]' [, ...])
+ | ('pxf://host[:port]/path-to-data<pxf parameters>')
+ FORMAT 'TEXT'
+ [( [HEADER]
+ [DELIMITER [AS] 'delimiter' | 'OFF']
+ [NULL [AS] 'null string']
+ [ESCAPE [AS] 'escape' | 'OFF']
+ [NEWLINE [ AS ] 'LF' | 'CR' | 'CRLF']
+ [FILL MISSING FIELDS] )]
+ | 'CSV'
+ [( [HEADER]
+ [QUOTE [AS] 'quote']
+ [DELIMITER [AS] 'delimiter']
+ [NULL [AS] 'null string']
+ [FORCE NOT NULL column [, ...]]
+ [ESCAPE [AS] 'escape']
+ [NEWLINE [ AS ] 'LF' | 'CR' | 'CRLF']
+ [FILL MISSING FIELDS] )]
+ | 'CUSTOM' (Formatter=<formatter specifications>)
+ [ ENCODING 'encoding' ]
+ [ [LOG ERRORS INTO error_table] SEGMENT REJECT LIMIT count
+ [ROWS | PERCENT] ]
+
+CREATE [READABLE] EXTERNAL WEB TABLE table_name
+ ( column_name
+ data_type [, ...] | LIKE other_table )
+ LOCATION ('http://webhost[:port]/path/file' [, ...])
+ | EXECUTE 'command' ON { MASTER | number_of_segments | SEGMENT #num }
+ FORMAT 'TEXT'
+ [( [HEADER]
+ [DELIMITER [AS] 'delimiter' | 'OFF']
+ [NULL [AS] 'null string']
+ [ESCAPE [AS] 'escape' | 'OFF']
+ [NEWLINE [ AS ] 'LF' | 'CR' | 'CRLF']
+ [FILL MISSING FIELDS] )]
+ | 'CSV'
+ [( [HEADER]
+ [QUOTE [AS] 'quote']
+ [DELIMITER [AS] 'delimiter']
+ [NULL [AS] 'null string']
+ [FORCE NOT NULL column [, ...]]
+ [ESCAPE [AS] 'escape']
+ [NEWLINE [ AS ] 'LF' | 'CR' | 'CRLF']
+ [FILL MISSING FIELDS] )]
+ | 'CUSTOM' (Formatter=<formatter specifications>)
+ [ ENCODING 'encoding' ]
+ [ [LOG ERRORS INTO error_table] SEGMENT REJECT LIMIT count
+ [ROWS | PERCENT] ]
+
+CREATE WRITABLE EXTERNAL TABLE table_name
+ ( column_name
+ data_type [, ...] | LIKE other_table )
+ LOCATION('gpfdist://outputhost[:port]/filename[#transform]'
+ | ('gpfdists://outputhost[:port]/file_pattern[#transform]'
+ [, ...])
+ | ('pxf://host[:port]/path-to-data?<pxf parameters>'
+ FORMAT 'TEXT'
+ [( [DELIMITER [AS] 'delimiter']
+ [NULL [AS] 'null string']
+ [ESCAPE [AS] 'escape' | 'OFF'] )]
+ | 'CSV'
+ [([QUOTE [AS] 'quote']
+ [DELIMITER [AS] 'delimiter']
+ [NULL [AS] 'null string']
+ [FORCE QUOTE column [, ...]] ]
+ [ESCAPE [AS] 'escape'] )]
+ | 'CUSTOM' (Formatter=<formatter specifications>)
+ [ ENCODING 'write_encoding' ]
+ [ DISTRIBUTED BY (column, [ ... ] ) | DISTRIBUTED RANDOMLY ]
+
+CREATE WRITABLE EXTERNAL WEB TABLE table_name
+ ( column_name
+ data_type [, ...] | LIKE other_table )
+ EXECUTE 'command' ON #num
+ FORMAT 'TEXT'
+ [( [DELIMITER [AS] 'delimiter']
+ [NULL [AS] 'null string']
+ [ESCAPE [AS] 'escape' | 'OFF'] )]
+ | 'CSV'
+ [([QUOTE [AS] 'quote']
+ [DELIMITER [AS] 'delimiter']
+ [NULL [AS] 'null string']
+ [FORCE QUOTE column [, ...]] ]
+ [ESCAPE [AS] 'escape'] )]
+ | 'CUSTOM' (Formatter=<formatter specifications>)
+ [ ENCODING 'write_encoding' ]
+ [ DISTRIBUTED BY (column, [ ... ] ) | DISTRIBUTED RANDOMLY ]
+```
+
+where *<pxf parameters>* is:
+
+``` pre
+ ?FRAGMENTER=class&ACCESSOR=class&RESOLVER=class[&custom-option=value...]
+ | ?PROFILE=profile-name[&custom-option=value...]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`CREATE EXTERNAL TABLE` or `CREATE EXTERNAL WEB TABLE` creates a new readable external table definition in HAWQ. Readable external tables are typically used for fast, parallel data loading. Once an external table is defined, you can query its data directly (and in parallel) using SQL commands. For example, you can select, join, or sort external table data. You can also create views for external tables. DML operations (`UPDATE`, `INSERT`, `DELETE`, or` TRUNCATE`) are not allowed on readable external tables.
+
+`CREATE WRITABLE EXTERNAL TABLE` or `CREATE WRITABLE EXTERNAL WEB TABLE` creates a new writable external table definition in HAWQ. Writable external tables are typically used for unloading data from the database into a set of files or named pipes.
+
+Writable external web tables can also be used to output data to an executable program. Once a writable external table is defined, data can be selected from database tables and inserted into the writable external table. Writable external tables only allow `INSERT` operations – `SELECT`, `UPDATE`, `DELETE`, or `TRUNCATE` are not allowed.
+
+Regular readable external tables can access static flat files or, by using HAWQ Extensions Framework (PXF), data from other sources. PXF plug-ins are included for HDFS, HBase, and Hive tables. Custom plug-ins can be created for other external data sources using the PXF API.
+
+Web external tables access dynamic data sources – either on a web server or by executing OS commands or scripts.
+
+The LOCATION clause specifies the location of the external data. The location string begins with a protocol string that specifies the storage type and protocol used to access the data. The `gpfdist://` protocol specifies data files served by one or more instances of the Greenplum parallel file distribution server `gpfdist`. The `http://` protocol specifies one or more HTTP URLs and is used with web tables. The `pxf://` protocol specifies data accessed through the PXF service, which provides access to data in a Hadoop system. Using the PXF API, you can create PXF plug-ins to provide HAWQ access to any other data source.
+
+**Note:** The `file://` protocol is deprecated. Instead, use the `gpfdist://`, `gpfdists://`, or `pxf://` protocol, or the `COPY` command instead.
+
+The `FORMAT` clause is used to describe how external table files are formatted. Valid flat file formats, including files in HDFS, are delimited text (`TEXT`) and comma separated values (`CSV`) format for `gpfdist` protocols. If the data in the file does not use the default column delimiter, escape character, null string, and so on, you must specify the additional formatting options so that the data in the external file is read correctly by HAWQ.
+
+## Parameters<a id="topic1__section4"></a>
+
+READABLE | WRITABLE
+Specifiies the type of external table, readable being the default. Readable external tables are used for loading data into HAWQ. Writable external tables are used for unloading data.
+
+WEB
+Creates a readable or writable web external table definition in HAWQ. There are two forms of readable web external tables – those that access files via the `http://` protocol or those that access data by executing OS commands. Writable web external tables output data to an executable program that can accept an input stream of data. Web external tables are not rescannable during query execution.
+
+ *table\_name*
+The name of the new external table.
+
+ *column\_name*
+The name of a column to create in the external table definition. Unlike regular tables, external tables do not have column constraints or default values, so do not specify those.
+
+LIKE *other\_table*
+The `LIKE` clause specifies a table from which the new external table automatically copies all column names, data types and HAWQ distribution policy. If the original table specifies any column constraints or default column values, those will not be copied over to the new external table definition.
+
+ *data\_type*
+The data type of the column.
+
+LOCATION *('protocol://host\[:port\]/path/file' \[, ...\])*
+For readable external tables, specifies the URI of the external data source(s) to be used to populate the external table or web table. Regular readable external tables allow the `file`, `gpfdist`, and `pxf` protocols. Web external tables allow the `http` protocol. If `port` is omitted, the `http` and `gpfdist` protocols assume port `8080` and the `pxf` protocol assumes the `host` is a high availability nameservice string. If using the `gpfdist` protocol, the `path` is relative to the directory from which `gpfdist` is serving files (the directory specified when you started the `gpfdist` program). Also, the path can use wildcards (or other C-style pattern matching) in the file name part of the location to denote multiple files in a directory. For example:
+
+``` pre
+'gpfdist://filehost:8081/*'
+'gpfdist://masterhost/my_load_file'
+'http://intranet.mycompany.com/finance/expenses.csv'
+'pxf://mdw:41200/sales/*.csv?Profile=HDFS'
+```
+
+For writable external tables, specifies the URI location of the `gpfdist` process that will collect data output from the HAWQ segments and write it to the named file. The `path` is relative to the directory from which `gpfdist` is serving files (the directory specified when you started the `gpfdist` program). If multiple `gpfdist` locations are listed, the segments sending data will be evenly divided across the available output locations. For example:
+
+``` pre
+'gpfdist://outputhost:8081/data1.out',
+'gpfdist://outputhost:8081/data2.out'
+```
+
+With two `gpfdist` locations listed as in the above example, half of the segments would send their output data to the `data1.out` file and the other half to the `data2.out` file.
+
+For the `pxf` protocol, the `LOCATION` string specifies the host and port of the PXF service, the location of the data, and the PXF plug-ins (Java classes) used to convert the data between storage format and HAWQ format. If the `port` is omitted, the host is taken to be the logical name for the high availability name service and the port is the value of the `pxf_service_port` configuration variable, 51200 by default. The URL parameters `FRAGMENTER`, `ACCESSOR`, and `RESOLVER` are the names of PXF plug-ins (Java classes) that convert between the external data format and HAWQ data format. The `FRAGMENTER` parameter is only used with readable external tables. PXF allows combinations of these parameters to be configured as profiles so that a single `PROFILE` parameter can be specified to access external data, for example `?PROFILE=Hive`. Additional `custom-options` can be added to the LOCATION URI to further describe the external data format or storage options (see [Additional Options](../../pxf/HDFSFileDataPXF.html#additionaloptions)). For details about the plug-ins and profiles provided with PXF and information about creating custom plug-ins for other data sources see [Working with PXF and External Data](../../pxf/PivotalExtensionFrameworkPXF.html).
+
+EXECUTE *'command'* ON ...
+Allowed for readable web external tables or writable external tables only. For readable web external tables, specifies the OS command to be executed by the segment instances. The *command* can be a single OS command or a script. If the command executes a script, that script must reside in the same location on all of the segment hosts and be executable by the HAWQ superuser (`gpadmin`).
+
+For writable external tables, the *command* specified in the `EXECUTE` clause must be prepared to have data piped into it, as segments having data to send write their output to the specified program. HAWQ uses virtual elastic segments to run its queries.
+
+The `ON` clause is used to specify which segment instances will execute the given command. For writable external tables, only `ON` *number* is supported.
+
+**Note:** ON ALL/HOST is deprecated when creating a readable external table, as HAWQ cannot guarantee scheduling executors on a specific host. Instead, use `ON MASTER`, `ON number`, or `SEGMENT virtual_segment` to specify which segment instances will execute the command.
+
+- `ON MASTER` runs the command on the master host only.
+- `ON number` means the command will be executed by the specified number of virtual segments. The particular segments are chosen by the HAWQ system's Resource Manager at runtime.
+- `ON SEGMENT virtual_segment ` means the command will be executed only once by the specified segment.
+
+<!-- -->
+
+FORMAT 'TEXT | CSV' *(options)*
+Specifies the format of the external or web table data - either plain text (`TEXT`) or comma separated values (`CSV`) format.
+
+DELIMITER
+Specifies a single ASCII character that separates columns within each row (line) of data. The default is a tab character in `TEXT` mode, a comma in `CSV` mode. In `TEXT` mode for readable external tables, the delimiter can be set to `OFF` for special use cases in which unstructured data is loaded into a single-column table.
+
+NULL
+Specifies the string that represents a `NULL` value. The default is `\N` (backslash-N) in `TEXT` mode, and an empty value with no quotations in `CSV` mode. You might prefer an empty string even in `TEXT` mode for cases where you do not want to distinguish `NULL` values from empty strings. When using external and web tables, any data item that matches this string will be considered a `NULL` value.
+
+ESCAPE
+Specifies the single character that is used for C escape sequences (such as `\n`,`\t`,`\100`, and so on) and for escaping data characters that might otherwise be taken as row or column delimiters. Make sure to choose an escape character that is not used anywhere in your actual column data. The default escape character is a \\ (backslash) for text-formatted files and a `"` (double quote) for csv-formatted files, however it is possible to specify another character to represent an escape. It is also possible to disable escaping in text-formatted files by specifying the value `'OFF'` as the escape value. This is very useful for data such as text-formatted web log data that has many embedded backslashes that are not intended to be escapes.
+
+NEWLINE
+Specifies the newline used in your data files – `LF` (Line feed, 0x0A), `CR` (Carriage return, 0x0D), or `CRLF` (Carriage return plus line feed, 0x0D 0x0A). If not specified, a HAWQ segment will detect the newline type by looking at the first row of data it receives and using the first newline type encountered.
+
+HEADER
+For readable external tables, specifies that the first line in the data file(s) is a header row (contains the names of the table columns) and should not be included as data for the table. If using multiple data source files, all files must have a header row.
+
+**Note:** The `HEADER` formatting option is not allowed with PXF.
+For CSV files or other files that include a header line, use an error table instead of the `HEADER` formatting option.
+
+QUOTE
+Specifies the quotation character for `CSV` mode. The default is double-quote (`"`).
+
+FORCE NOT NULL
+In `CSV` mode, processes each specified column as though it were quoted and hence not a `NULL` value. For the default null string in `CSV` mode (nothing between two delimiters), this causes missing values to be evaluated as zero-length strings.
+
+FORCE QUOTE
+In `CSV` mode for writable external tables, forces quoting to be used for all non-`NULL` values in each specified column. `NULL` output is never quoted.
+
+FILL MISSING FIELDS
+In both `TEXT` and `CSV` mode for readable external tables, specifying `FILL MISSING FIELDS` will set missing trailing field values to `NULL` (instead of reporting an error) when a row of data has missing data fields at the end of a line or row. Blank rows, fields with a `NOT NULL` constraint, and trailing delimiters on a line will still report an error.
+
+ENCODING *'encoding'*
+Character set encoding to use for the external table. Specify a string constant (such as `'SQL_ASCII'`), an integer encoding number, or `DEFAULT` to use the default client encoding.
+
+LOG ERRORS INTO *error\_table*
+This is an optional clause that can precede a `SEGMENT REJECT LIMIT` clause to log information about rows with formatting errors. It specifies an error table where rows with formatting errors will be logged when running in single row error isolation mode. You can then examine this error table to see error rows that were not loaded (if any). If the error\_table specified already exists, it will be used. If it does not exist, it will be automatically generated.
+
+SEGMENT REJECT LIMIT *count* \[ROWS | PERCENT\]
+Runs a `COPY FROM` operation in single row error isolation mode. If the input rows have format errors they will be discarded provided that the reject limit count is not reached on any HAWQ segment instance during the load operation. The reject limit count can be specified as number of rows (the default) or percentage of total rows (1-100). If `PERCENT` is used, each segment starts calculating the bad row percentage only after the number of rows specified by the parameter `gp_reject_percent_threshold` has been processed. The default for `gp_reject_percent_threshold` is 300 rows. Constraint errors such as violation of a `NOT NULL` or `CHECK` constraint will still be handled in "all-or-nothing" input mode. If the limit is not reached, all good rows will be loaded and any error rows discarded.
+
+DISTRIBUTED RANDOMLY
+Used to declare the HAWQ distribution policy for a writable external table. By default, writable external tables are distributed randomly. If the source table you are exporting data from has a hash distribution policy, defining the same distribution key column(s) for the writable external table will improve unload performance by eliminating the need to move rows over the interconnect. When you issue an unload command such as `INSERT INTO wex_table SELECT * FROM source_table `, the rows that are unloaded can be sent directly from the segments to the output location if the two tables have the same hash distribution policy.
+
+## Examples<a id="topic1__section5"></a>
+
+Start the `gpfdist` file server program in the background on port `8081` serving files from directory `/var/data/staging`:
+
+``` pre
+gpfdist -p 8081 -d /var/data/staging -l /home/gpadmin/log &
+```
+
+Create a readable external table named `ext_customer` using the `gpfdist` protocol and any text formatted files (`*.txt`) found in the `gpfdist` directory. The files are formatted with a pipe (`|`) as the column delimiter and an empty space as `NULL`. Also access the external table in single row error isolation mode:
+
+``` pre
+CREATE EXTERNAL TABLE ext_customer
+ (id int, name text, sponsor text)
+ LOCATION ( 'gpfdist://filehost:8081/*.txt' )
+ FORMAT 'TEXT' ( DELIMITER '|' NULL ' ')
+ LOG ERRORS INTO err_customer SEGMENT REJECT LIMIT 5;
+```
+
+Create the same readable external table definition as above, but with CSV formatted files:
+
+``` pre
+CREATE EXTERNAL TABLE ext_customer
+ (id int, name text, sponsor text)
+ LOCATION ( 'gpfdist://filehost:8081/*.csv' )
+ FORMAT 'CSV' ( DELIMITER ',' );
+```
+
+Create a readable external table using the `pxf` protocol to read data in HDFS files:
+
+``` pre
+CREATE EXTERNAL TABLE ext_customer
+ (id int, name text, sponsor text)
+LOCATION ('pxf://mdw:51200/sales/customers/customers.tsv.gz'
+ '?Fragmenter=org.apache.hawq.pxf.plugins.hdfs.HdfsDataFragmenter'
+ '&Accessor=org.apache.hawq.pxf.plugins.hdfs.LineBreakAccessor'
+ '&Resolver=org.apache.hawq.pxf.plugins.hdfs.StringPassResolver')
+FORMAT 'TEXT' (DELIMITER = E'\t');
+```
+
+The `LOCATION` string in this command is equivalent to the previous example, but using a PXF Profile:
+
+``` pre
+CREATE EXTERNAL TABLE ext_customer
+ (id int, name text, sponsor text)
+LOCATION ('pxf://mdw:51200/sales/customers/customers.tsv.gz?Profile=HdfsTextSimple')
+FORMAT 'TEXT' (DELIMITER = E'\t');
+```
+
+Create a readable web external table that executes a script on five virtual segment hosts. (The script must reside at the same location on all segment hosts.)
+
+``` pre
+CREATE EXTERNAL WEB TABLE log_output (linenum int, message text)
+EXECUTE '/var/load_scripts/get_log_data.sh' ON 5
+FORMAT 'TEXT' (DELIMITER '|');
+```
+
+Create a writable external table named `sales_out` that uses `gpfdist` to write output data to a file named `sales.out`. The files are formatted with a pipe (`|`) as the column delimiter and an empty space as `NULL`.
+
+``` pre
+CREATE WRITABLE EXTERNAL TABLE sales_out (LIKE sales)
+ LOCATION ('gpfdist://etl1:8081/sales.out')
+ FORMAT 'TEXT' ( DELIMITER '|' NULL ' ')
+ DISTRIBUTED BY (txn_id);
+```
+
+The following command sequence shows how to create a writable external web table using a specified number of elastic virtual segments to run the query:
+
+``` pre
+postgres=# CREATE TABLE a (i int);
+CREATE TABLE
+postgres=# INSERT INTO a VALUES(1);
+INSERT 0 1
+postgres=# INSERT INTO a VALUES(2);
+INSERT 0 1
+postgres=# INSERT INTO a VALUES(10);
+INSERT 0 1
+postgres=# CREATE WRITABLE EXTERNAL WEB TABLE externala (output text)
+postgres-# EXECUTE 'cat > /tmp/externala' ON 3
+postgres-# FORMAT 'TEXT' DISTRIBUTED RANDOMLY;
+CREATE EXTERNAL TABLE
+postgres=# INSERT INTO externala SELECT * FROM a;
+INSERT 0 3
+```
+
+Create a writable external web table that pipes output data received by the segments to an executable script named `to_adreport_etl.sh`:
+
+``` pre
+CREATE WRITABLE EXTERNAL WEB TABLE campaign_out (LIKE campaign)
+EXECUTE '/var/unload_scripts/to_adreport_etl.sh'
+FORMAT 'TEXT' (DELIMITER '|');
+```
+
+Use the writable external table defined above to unload selected data:
+
+``` pre
+INSERT INTO campaign_out
+ SELECT * FROM campaign WHERE customer_id=123;
+```
+
+## Compatibility<a id="topic1__section6"></a>
+
+`CREATE EXTERNAL TABLE` is a HAWQ extension. The SQL standard makes no provisions for external tables.
+
+## See Also<a id="topic1__section7"></a>
+
+[CREATE TABLE](CREATE-TABLE.html), [CREATE TABLE AS](CREATE-TABLE-AS.html), [COPY](COPY.html), [INSERT](INSERT.html), [SELECT INTO](SELECT-INTO.html)
diff --git a/reference/sql/CREATE-FUNCTION.html.md.erb b/reference/sql/CREATE-FUNCTION.html.md.erb
new file mode 100644
index 0000000..9dd6f38
--- /dev/null
+++ b/reference/sql/CREATE-FUNCTION.html.md.erb
@@ -0,0 +1,190 @@
+---
+title: CREATE FUNCTION
+---
+
+Defines a new function.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+CREATE [OR REPLACE] FUNCTION name
+ ( [ [argmode] [argname] argtype [, ...] ] )
+ [ RETURNS { [ SETOF ] rettype
+ | TABLE ([{ argname argtype | LIKE other table }
+ [, ...]])
+ } ]
+ { LANGUAGE langname
+ | IMMUTABLE | STABLE | VOLATILE
+ | CALLED ON NULL INPUT | RETURNS NULL ON NULL INPUT | STRICT
+ | [EXTERNAL] SECURITY INVOKER | [EXTERNAL] SECURITY DEFINER
+ | AS 'definition'
+ | AS 'obj_file', 'link_symbol' } ...
+ [ WITH ({ DESCRIBE = describe_function
+ } [, ...] ) ]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`CREATE FUNCTION` defines a new function. `CREATE OR REPLACE FUNCTION` will either create a new function, or replace an existing definition.
+
+The name of the new function must not match any existing function with the same argument types in the same schema. However, functions of different argument types may share a name (overloading).
+
+To update the definition of an existing function, use `CREATE OR REPLACE FUNCTION`. It is not possible to change the name or argument types of a function this way (this would actually create a new, distinct function). Also, `CREATE OR REPLACE FUNCTION` will not let you change the return type of an existing function. To do that, you must drop and recreate the function. If you drop and then recreate a function, you will have to drop existing objects (rules, views, and so on) that refer to the old function. Use `CREATE OR REPLACE FUNCTION` to change a function definition without breaking objects that refer to the function.
+
+For more information about creating functions, see [User-Defined Functions](../../query/functions-operators.html#topic28).
+
+**Limited Use of VOLATILE and STABLE Functions**
+
+To prevent data from becoming out-of-sync across the segments in HAWQ, any function classified as `STABLE` or `VOLATILE` cannot be executed at the segment level if it contains SQL or modifies the database in any way. For example, functions such as `random()` or `timeofday()` are not allowed to execute on distributed data in HAWQ because they could potentially cause inconsistent data between the segment instances.
+
+To ensure data consistency, `VOLATILE` and `STABLE` functions can safely be used in statements that are evaluated on and execute from the master. For example, the following statements are always executed on the master (statements without a `FROM` clause):
+
+``` pre
+SELECT setval('myseq', 201);
+SELECT foo();
+```
+
+In cases where a statement has a `FROM` clause containing a distributed table and the function used in the `FROM` clause simply returns a set of rows, execution may be allowed on the segments:
+
+``` pre
+SELECT * FROM foo();
+```
+
+One exception to this rule are functions that return a table reference (`rangeFuncs`) or functions that use the `refCursor` data type. Note that you cannot return a `refcursor` from any kind of function in HAWQ.
+
+## Parameters<a id="topic1__section5"></a>
+
+ *name*
+The name (optionally schema-qualified) of the function to create.
+
+ *argmode*
+The mode of an argument: either `IN`, `OUT`, or `INOUT`. If omitted, the default is `IN`.
+
+ *argname*
+The name of an argument. Some languages (currently only PL/pgSQL) let you use the name in the function body. For other languages the name of an input argument is just extra documentation. But the name of an output argument is significant, since it defines the column name in the result row type. (If you omit the name for an output argument, the system will choose a default column name.)
+
+ *argtype*
+The data type(s) of the function's arguments (optionally schema-qualified), if any. The argument types may be base, composite, or domain types, or may reference the type of a table column.
+
+Depending on the implementation language it may also be allowed to specify pseudotypes such as `cstring`. Pseudotypes indicate that the actual argument type is either incompletely specified, or outside the set of ordinary SQL data types.
+
+The type of a column is referenced by writing ` tablename.columnname%TYPE`. Using this feature can sometimes help make a function independent of changes to the definition of a table.
+
+ *rettype*
+The return data type (optionally schema-qualified). The return type can be a base, composite, or domain type, or may reference the type of a table column. Depending on the implementation language it may also be allowed to specify pseudotypes such as `cstring`. If the function is not supposed to return a value, specify `void` as the return type.
+
+When there are `OUT` or `INOUT` parameters, the `RETURNS` clause may be omitted. If present, it must agree with the result type implied by the output parameters: `RECORD` if there are multiple output parameters, or the same type as the single output parameter.
+
+The `SETOF` modifier indicates that the function will return a set of items, rather than a single item.
+
+The type of a column is referenced by writing ` tablename.columnname%TYPE`.
+
+ *langname*
+The name of the language that the function is implemented in. May be `SQL`, `C`, `internal`, or the name of a user-defined procedural language. See [CREATE LANGUAGE](CREATE-LANGUAGE.html) for the procedural languages supported in HAWQ. For backward compatibility, the name may be enclosed by single quotes.
+
+IMMUTABLE
+STABLE
+VOLATILE
+These attributes inform the query optimizer about the behavior of the function. At most one choice may be specified. If none of these appear, `VOLATILE` is the default assumption. Since HAWQ currently has limited use of `VOLATILE` functions, if a function is truly `IMMUTABLE`, you must declare it as so to be able to use it without restrictions.
+
+`IMMUTABLE` indicates that the function cannot modify the database and always returns the same result when given the same argument values. It does not do database lookups or otherwise use information not directly present in its argument list. If this option is given, any call of the function with all-constant arguments can be immediately replaced with the function value.
+
+`STABLE` indicates that the function cannot modify the database, and that within a single table scan it will consistently return the same result for the same argument values, but that its result could change across SQL statements. This is the appropriate selection for functions whose results depend on database lookups, parameter values (such as the current time zone), and so on. Also note that the *current\_timestamp* family of functions qualify as stable, since their values do not change within a transaction.
+
+`VOLATILE` indicates that the function value can change even within a single table scan, so no optimizations can be made. Relatively few database functions are volatile in this sense; some examples are `random()`, `currval()`, `timeofday()`. But note that any function that has side-effects must be classified volatile, even if its result is quite predictable, to prevent calls from being optimized away; an example is `setval()`.
+
+CALLED ON NULL INPUT
+RETURNS NULL ON NULL INPUT
+STRICT
+`CALLED ON NULL INPUT` (the default) indicates that the function will be called normally when some of its arguments are null. It is then the function author's responsibility to check for null values if necessary and respond appropriately. `RETURNS NULL ON NULL INPUT` or `STRICT` indicates that the function always returns null whenever any of its arguments are null. If this parameter is specified, the function is not executed when there are null arguments; instead a null result is assumed automatically.
+
+\[EXTERNAL\] SECURITY INVOKER
+\[EXTERNAL\] SECURITY DEFINER
+`SECURITY INVOKER` (the default) indicates that the function is to be executed with the privileges of the user that calls it. `SECURITY DEFINER` specifies that the function is to be executed with the privileges of the user that created it. The key word `EXTERNAL` is allowed for SQL conformance, but it is optional since, unlike in SQL, this feature applies to all functions not just external ones.
+
+ *definition*
+A string constant defining the function; the meaning depends on the language. It may be an internal function name, the path to an object file, an SQL command, or text in a procedural language.
+
+ *obj\_file, link\_symbol*
+This form of the `AS` clause is used for dynamically loadable C language functions when the function name in the C language source code is not the same as the name of the SQL function. The string *obj\_file* is the name of the file containing the dynamically loadable object, and *link\_symbol* is the name of the function in the C language source code. If the link symbol is omitted, it is assumed to be the same as the name of the SQL function being defined. A good practice is to locate shared libraries either relative to `$libdir` (which is located at `$GPHOME/lib`) or through the dynamic library path (set by the `dynamic_library_path` server configuration parameter). This simplifies version upgrades if the new installation is at a different location.
+
+ *describe\_function*
+The name of a callback function to execute when a query that calls this function is parsed. The callback function returns a tuple descriptor that indicates the result type.
+
+## Notes<a id="topic1__section6"></a>
+
+Any compiled code (shared library files) for custom functions must be placed in the same location on every host in your HAWQ array (master and all segments). This location must also be in the `LD_LIBRARY_PATH` so that the server can locate the files. Consider locating shared libraries either relative to `$libdir` (which is located at `$GPHOME/lib`) or through the dynamic library path (set by the `dynamic_library_path` server configuration parameter) on all master segment instances in the HAWQ array.
+
+The full SQL type syntax is allowed for input arguments and return value. However, some details of the type specification (such as the precision field for type *numeric*) are the responsibility of the underlying function implementation and are not recognized or enforced by the `CREATE FUNCTION` command.
+
+HAWQ allows function overloading. The same name can be used for several different functions so long as they have distinct argument types. However, the C names of all functions must be different, so you must give overloaded C functions different C names (for example, use the argument types as part of the C names).
+
+Two functions are considered the same if they have the same names and input argument types, ignoring any `OUT` parameters. Thus for example these declarations conflict:
+
+``` pre
+CREATE FUNCTION foo(int) ...
+CREATE FUNCTION foo(int, out text) ...
+```
+
+When repeated `CREATE FUNCTION` calls refer to the same object file, the file is only loaded once. To unload and reload the file, use the `LOAD` command.
+
+To be able to define a function, the user must have the `USAGE` privilege on the language.
+
+It is often helpful to use dollar quoting to write the function definition string, rather than the normal single quote syntax. Without dollar quoting, any single quotes or backslashes in the function definition must be escaped by doubling them. A dollar-quoted string constant consists of a dollar sign (`$`), an optional tag of zero or more characters, another dollar sign, an arbitrary sequence of characters that makes up the string content, a dollar sign, the same tag that began this dollar quote, and a dollar sign. Inside the dollar-quoted string, single quotes, backslashes, or any character can be used without escaping. The string content is always written literally. For example, here are two different ways to specify the string "Dianne's horse" using dollar quoting:
+
+``` pre
+$$Dianne's horse$$
+$SomeTag$Dianne's horse$SomeTag$
+```
+
+## Examples<a id="topic1__section8"></a>
+
+A very simple addition function:
+
+``` pre
+CREATE FUNCTION add(integer, integer) RETURNS integer
+ AS 'select $1 + $2;'
+ LANGUAGE SQL
+ IMMUTABLE
+ RETURNS NULL ON NULL INPUT;
+```
+
+Increment an integer, making use of an argument name, in PL/pgSQL:
+
+``` pre
+CREATE OR REPLACE FUNCTION increment(i integer) RETURNS
+integer AS $$
+ BEGIN
+ RETURN i + 1;
+ END;
+$$ LANGUAGE plpgsql;
+```
+
+Return a record containing multiple output parameters:
+
+``` pre
+CREATE FUNCTION dup(in int, out f1 int, out f2 text)
+ AS $$ SELECT $1, CAST($1 AS text) || ' is text' $$
+ LANGUAGE SQL;
+SELECT * FROM dup(42);
+```
+
+You can do the same thing more verbosely with an explicitly named composite type:
+
+``` pre
+CREATE TYPE dup_result AS (f1 int, f2 text);
+CREATE FUNCTION dup(int) RETURNS dup_result
+ AS $$ SELECT $1, CAST($1 AS text) || ' is text' $$
+ LANGUAGE SQL;
+SELECT * FROM dup(42);
+```
+
+## Compatibility<a id="topic1__section9"></a>
+
+`CREATE FUNCTION` is defined in SQL:1999 and later. The HAWQ version is similar, but not fully compatible. The attributes are not portable, neither are the different available languages.
+
+For compatibility with some other database systems, *argmode* can be written either before or after *argname*. But only the first way is standard-compliant.
+
+## See Also<a id="topic1__section10"></a>
+
+[ALTER FUNCTION](ALTER-FUNCTION.html), [DROP FUNCTION](DROP-FUNCTION.html)
diff --git a/reference/sql/CREATE-GROUP.html.md.erb b/reference/sql/CREATE-GROUP.html.md.erb
new file mode 100644
index 0000000..1e62ade
--- /dev/null
+++ b/reference/sql/CREATE-GROUP.html.md.erb
@@ -0,0 +1,43 @@
+---
+title: CREATE GROUP
+---
+
+Defines a new database role.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+CREATE GROUP name [ [WITH] option [ ... ] ]
+```
+
+where *option* can be:
+
+``` pre
+ SUPERUSER | NOSUPERUSER
+ | CREATEDB | NOCREATEDB
+ | CREATEROLE | NOCREATEROLE
+ | CREATEUSER | NOCREATEUSER
+ | INHERIT | NOINHERIT
+ | LOGIN | NOLOGIN
+ | [ ENCRYPTED | UNENCRYPTED ] PASSWORD 'password'
+ | VALID UNTIL 'timestamp'
+ | IN ROLE rolename [, ...]
+ | IN GROUP rolename [, ...]
+ | ROLE rolename [, ...]
+ | ADMIN rolename [, ...]
+ | USER rolename [, ...]
+ | SYSID uid
+
+```
+
+## Description<a id="topic1__section3"></a>
+
+`CREATE GROUP` has been replaced by [CREATE ROLE](CREATE-ROLE.html), although it is still accepted for backwards compatibility.
+
+## Compatibility<a id="topic1__section4"></a>
+
+There is no `CREATE GROUP` statement in the SQL standard.
+
+## See Also<a id="topic1__section5"></a>
+
+[CREATE ROLE](CREATE-ROLE.html)
diff --git a/reference/sql/CREATE-LANGUAGE.html.md.erb b/reference/sql/CREATE-LANGUAGE.html.md.erb
new file mode 100644
index 0000000..986908d
--- /dev/null
+++ b/reference/sql/CREATE-LANGUAGE.html.md.erb
@@ -0,0 +1,97 @@
+---
+title: CREATE LANGUAGE
+---
+
+Defines a new procedural language.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+CREATE [PROCEDURAL] LANGUAGE name
+
+CREATE [TRUSTED] [PROCEDURAL] LANGUAGE name
+ HANDLER call_handler [VALIDATOR valfunction]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`CREATE LANGUAGE` registers a new procedural language with a HAWQ database. Subsequently, functions can be defined in this new language. You must be a superuser to register a new language.
+
+When you register a new procedural language, you effectively associate the language name with a call handler that is responsible for executing functions written in that language. For a function written in a procedural language (a language other than C or SQL), the database server has no built-in knowledge about how to interpret the function's source code. The task is passed to a special handler that knows the details of the language. The handler could either do all the work of parsing,syntax analysis, execution, and so on, or it could serve as a bridge between HAWQ and an existing implementation of a programming language. The handler itself is a C language function compiled into a shared object and loaded on demand, just like any other C function.
+
+There are two forms of the `CREATE LANGUAGE` command. In the first form, the user specifies the name of the desired language and the HAWQ server uses the `pg_pltemplate` system catalog to determine the correct parameters. In the second form, the user specifies the language parameters as well as the language name. You can use the second form to create a language that is not defined in `pg_pltemplate`.
+
+When the server finds an entry in the `pg_pltemplate` catalog for the given language name, it will use the catalog data even if the command includes language parameters. This behavior simplifies loading of old dump files, which are likely to contain out-of-date information about language support functions.
+
+## Parameters<a id="topic1__section4"></a>
+
+TRUSTED
+Ignored if the server has an entry for the specified language name in *pg\_pltemplate*. Specifies that the call handler for the language is safe and does not offer an unprivileged user any functionality to bypass access restrictions. If this key word is omitted when registering the language, only users with the superuser privilege can use this language to create new functions.
+
+PROCEDURAL
+Indicates that this is a procedural language.
+
+ *name*
+The name of the new procedural language. The language name is case insensitive. The name must be unique among the languages in the database. Built-in support is included for `plpgsql`, `plpython`, `plpythonu`, and `plr`. `plpgsql` is already installed by default in HAWQ.
+
+HANDLER *call\_handler*
+Ignored if the server has an entry for the specified language name in `pg_pltemplate`. The name of a previously registered function that will be called to execute the procedural language functions. The call handler for a procedural language must be written in a compiled language such as C with version 1 call convention and registered with HAWQ as a function taking no arguments and returning the `language_handler` type, a placeholder type that is simply used to identify the function as a call handler.
+
+VALIDATOR *valfunction*
+Ignored if the server has an entry for the specified language name in `pg_pltemplate`. *valfunction* is the name of a previously registered function that will be called when a new function in the language is created, to validate the new function. If no validator function is specified, then a new function will not be checked when it is created. The validator function must take one argument of type `oid`, which will be the OID of the to-be-created function, and will typically return `void`.
+
+A validator function would typically inspect the function body for syntactical correctness, but it can also look at other properties of the function, for example if the language cannot handle certain argument types. To signal an error, the validator function should use the `ereport()` function. The return value of the function is ignored.
+
+## Notes<a id="topic1__section5"></a>
+
+The procedural language packages included in the standard HAWQ distribution are:
+
+- `PL/pgSQL`
+- `PL/Perl`
+- `PL/Python`
+- `PL/Java`
+
+The `PL/pgSQL` language is registered in all databases by default. HAWQ has a language handler for `PL/R`, but the `PL/R` language package is not pre-installed with HAWQ.
+
+The PL/R libraries require the correct versions of R to be installed, respectively. Download the appropriate extensions from [Pivotal Network](https://network.pivotal.io/), then install the extensions. This ensures that all underlying dependencies are installed along with the extensions.
+
+To use the PL/R procedural language you must make sure that the systems that run HAWQ (master and all segments) have the R language installed and the PL/R package library (`plr.so`) added to their HAWQ installation on all hosts. Compiled packages for R and PL/R are available to install.
+
+The system catalog `pg_language` records information about the currently installed languages.
+
+To create functions in a procedural language, a user must have the `USAGE` privilege for the language. By default, `USAGE` is granted to `PUBLIC` (everyone) for trusted languages. This may be revoked if desired.
+
+Procedural languages are local to individual databases. However, a language can be installed into the `template1` database, which will cause it to be available automatically in all subsequently-created databases.
+
+The call handler function and the validator function (if any) must already exist if the server does not have an entry for the language in `pg_pltemplate`. But when there is an entry, the functions need not already exist; they will be automatically defined if not present in the database.
+
+Any shared library that implements a language must be located in the same `LD_LIBRARY_PATH` location on all segment hosts in your HAWQ array.
+
+## Examples<a id="topic1__section6"></a>
+
+The preferred way of creating any of the standard procedural languages in a database:
+
+``` pre
+CREATE LANGUAGE plr;
+CREATE LANGUAGE plpythonu;
+CREATE LANGUAGE plperl;
+```
+
+For a language not known in the `pg_pltemplate` catalog:
+
+``` pre
+CREATE FUNCTION plsample_call_handler() RETURNS
+language_handler
+ AS '$libdir/plsample'
+ LANGUAGE C;
+CREATE LANGUAGE plsample
+ HANDLER plsample_call_handler;
+```
+
+## Compatibility<a id="topic1__section7"></a>
+
+`CREATE LANGUAGE` is a HAWQ extension.
+
+## See Also<a id="topic1__section8"></a>
+
+[CREATE FUNCTION](CREATE-FUNCTION.html)
diff --git a/reference/sql/CREATE-OPERATOR-CLASS.html.md.erb b/reference/sql/CREATE-OPERATOR-CLASS.html.md.erb
new file mode 100644
index 0000000..cdc0c0d
--- /dev/null
+++ b/reference/sql/CREATE-OPERATOR-CLASS.html.md.erb
@@ -0,0 +1,151 @@
+---
+title: CREATE OPERATOR CLASS
+---
+
+Defines a new operator class.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+CREATE OPERATOR CLASS name [DEFAULT] FOR TYPE data_type
+ USING index_method AS
+ {
+ OPERATOR strategy_number
+ op_name [(op_type, op_type)] [RECHECK]
+ | FUNCTION support_number
+ funcname (argument_type [, ...] )
+ | STORAGE storage_type
+ } [, ... ]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`CREATE OPERATOR CLASS` creates a new operator class. An operator class defines how a particular data type can be used with an index. The operator class specifies that certain operators will fill particular roles or strategies for this data type and this index method. The operator class also specifies the support procedures to be used by the index method when the operator class is selected for an index column. All the operators and functions used by an operator class must be defined before the operator class is created. Any functions used to implement the operator class must be defined as `IMMUTABLE`.
+
+`CREATE OPERATOR CLASS` does not presently check whether the operator class definition includes all the operators and functions required by the index method, nor whether the operators and functions form a self-consistent set. It is the user's responsibility to define a valid operator class.
+
+You must be a superuser to create an operator class.
+
+## Parameters<a id="topic1__section4"></a>
+
+ *name*
+The (optionally schema-qualified) name of the operator class to be defined. Two operator classes in the same schema can have the same name only if they are for different index methods.
+
+DEFAULT
+Makes the operator class the default operator class for its data type. At most one operator class can be the default for a specific data type and index method.
+
+ *data\_type*
+The column data type that this operator class is for.
+
+ *index\_method*
+The name of the index method this operator class is for. Choices are `btree`, `bitmap`, and `gist`.
+
+ *strategy\_number*
+The operators associated with an operator class are identified by *strategy numbers*, which serve to identify the semantics of each operator within the context of its operator class. For example, B-trees impose a strict ordering on keys, lesser to greater, and so operators like *less than* and *greater than or equal to* are interesting with respect to a B-tree. These strategies can be thought of as generalized operators. Each operator class specifies which actual operator corresponds to each strategy for a particular data type and interpretation of the index semantics. The corresponding strategy numbers for each index method are as follows: <a id="topic1__bx145491"></a>
+
+<span class="tablecap">Table 1. B-tree and Bitmap Strategies</span>
+
+| Operation | Strategy Number |
+|-----------------------|-----------------|
+| less than | 1 |
+| less than or equal | 2 |
+| equal | 3 |
+| greater than or equal | 4 |
+| greater than | 5 |
+
+<span class="tablecap">Table 2. GiST Two-Dimensional Strategies (R-Tree)</span>
+
+<a id="topic1__bx145491a"></a>
+
+| Operation | Strategy Number |
+|-----------------------------|-----------------|
+| strictly left of | 1 |
+| does not extend to right of | 2 |
+| overlaps | 3 |
+| does not extend to left of | 4 |
+| strictly right of | 5 |
+| same | 6 |
+| contains | 7 |
+| contained by | 8 |
+| does not extend above | 9 |
+| strictly below | 10 |
+| strictly above | 11 |
+
+ *operator\_name*
+The name (optionally schema-qualified) of an operator associated with the operator class.
+
+ *op\_type*
+The operand data type(s) of an operator, or `NONE` to signify a left-unary or right-unary operator. The operand data types may be omitted in the normal case where they are the same as the operator class data type.
+
+RECHECK
+If present, the index is "lossy" for this operator, and so the rows retrieved using the index must be rechecked to verify that they actually satisfy the qualification clause involving this operator.
+
+ *support\_number*
+Index methods require additional support routines in order to work. These operations are administrative routines used internally by the index methods. As with strategies, the operator class identifies which specific functions should play each of these roles for a given data type and semantic interpretation. The index method defines the set of functions it needs, and the operator class identifies the correct functions to use by assigning them to the *support function numbers* as follows: <a id="topic1__bx145974"></a>
+
+<span class="tablecap">Table 3. B-tree and Bitmap Support Functions</span>
+
+| Function | Support Number |
+|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------|
+| Compare two keys and return an integer less than zero, zero, or greater than zero, indicating whether the first key is less than, equal to, or greater than the second. | 1 |
+
+
+<span class="tablecap">Table 4. GiST Support Functions</span>
+
+<a id="topic1__bx145974a"></a>
+
+| Function | Support Number |
+|-------------------------------------------------------------------------------------------------------------------------------|----------------|
+| consistent - determine whether key satisfies the query qualifier. | 1 |
+| union - compute union of a set of keys. | 2 |
+| compress - compute a compressed representation of a key or value to be indexed. | 3 |
+| decompress - compute a decompressed representation of a compressed key. | 4 |
+| penalty - compute penalty for inserting new key into subtree with given subtree's key. | 5 |
+| picksplit - determine which entries of a page are to be moved to the new page and compute the union keys for resulting pages. | 6 |
+| equal - compare two keys and return true if they are equal. | 7 |
+
+ *funcname*
+The name (optionally schema-qualified) of a function that is an index method support procedure for the operator class.
+
+ *argument\_types*
+The parameter data type(s) of the function.
+
+ *storage\_type*
+The data type actually stored in the index. Normally this is the same as the column data type, but the GiST index method allows it to be different. The `STORAGE` clause must be omitted unless the index method allows a different type to be used.
+
+## Notes<a id="topic1__section5"></a>
+
+Because the index machinery does not check access permissions on functions before using them, including a function or operator in an operator class is the same as granting public execute permission on it. This is usually not an issue for the sorts of functions that are useful in an operator class.
+
+The operators should not be defined by SQL functions. A SQL function is likely to be inlined into the calling query, which will prevent the optimizer from recognizing that the query matches an index.
+
+Any functions used to implement the operator class must be defined as `IMMUTABLE`.
+
+## Examples<a id="topic1__section6"></a>
+
+The following example command defines a GiST index operator class for the data type `_int4` (array of int4):
+
+``` pre
+CREATE OPERATOR CLASS gist__int_ops
+ DEFAULT FOR TYPE _int4 USING gist AS
+ OPERATOR 3 &&,
+ OPERATOR 6 = RECHECK,
+ OPERATOR 7 @>,
+ OPERATOR 8 <@,
+ OPERATOR 20 @@ (_int4, query_int),
+ FUNCTION 1 g_int_consistent (internal, _int4, int4),
+ FUNCTION 2 g_int_union (bytea, internal),
+ FUNCTION 3 g_int_compress (internal),
+ FUNCTION 4 g_int_decompress (internal),
+ FUNCTION 5 g_int_penalty (internal, internal, internal),
+ FUNCTION 6 g_int_picksplit (internal, internal),
+ FUNCTION 7 g_int_same (_int4, _int4, internal);
+```
+
+## Compatibility<a id="topic1__section7"></a>
+
+`CREATE OPERATOR CLASS` is a HAWQ extension. There is no `CREATE OPERATOR CLASS` statement in the SQL standard.
+
+## See Also<a id="topic1__section8"></a>
+
+[ALTER OPERATOR CLASS](ALTER-OPERATOR-CLASS.html), [DROP OPERATOR CLASS](DROP-OPERATOR-CLASS.html), [CREATE FUNCTION](CREATE-FUNCTION.html)
diff --git a/reference/sql/CREATE-OPERATOR.html.md.erb b/reference/sql/CREATE-OPERATOR.html.md.erb
new file mode 100644
index 0000000..cce7260
--- /dev/null
+++ b/reference/sql/CREATE-OPERATOR.html.md.erb
@@ -0,0 +1,168 @@
+---
+title: CREATE OPERATOR
+---
+
+Defines a new operator.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+CREATE OPERATOR name (
+ PROCEDURE = funcname
+ [, LEFTARG = lefttype] [, RIGHTARG = righttype]
+ [, COMMUTATOR = com_op] [, NEGATOR = neg_op]
+ [, RESTRICT = res_proc] [, JOIN = join_proc]
+ [, HASHES] [, MERGES]
+ [, SORT1 = left_sort_op] [, SORT2 = right_sort_op]
+ [, LTCMP = less_than_op] [, GTCMP = greater_than_op] )
+```
+
+## Description<a id="topic1__section3"></a>
+
+`CREATE OPERATOR` defines a new operator. The user who defines an operator becomes its owner.
+
+The operator name is a sequence of up to `NAMEDATALEN`-1 (63 by default) characters from the following list: `` + - * / < > = ~ ! @ # % ^ & | ` ? ``
+
+There are a few restrictions on your choice of name:
+
+- `--` and `/*` cannot appear anywhere in an operator name, since they will be taken as the start of a comment.
+- A multicharacter operator name cannot end in `+` or `-`, unless the name also contains at least one of these characters: `` ~ ! @ # % ^ & | ` ? ``
+
+For example, `@-` is an allowed operator name, but `*-` is not. This restriction allows HAWQ to parse SQL-compliant commands without requiring spaces between tokens.
+
+The operator `!=` is mapped to `<>` on input, so these two names are always equivalent.
+
+At least one of `LEFTARG` and `RIGHTARG` must be defined. For binary operators, both must be defined. For right unary operators, only `LEFTARG` should be defined, while for left unary operators only `RIGHTARG` should be defined.
+
+The *funcname* procedure must have been previously defined using `CREATE FUNCTION`, must be `IMMUTABLE`, and must be defined to accept the correct number of arguments (either one or two) of the indicated types.
+
+The other clauses specify optional operator optimization clauses. These clauses should be provided whenever appropriate to speed up queries that use the operator. But if you provide them, you must be sure that they are correct. Incorrect use of an optimization clause can result in server process crashes, subtly wrong output, or other unexpected results. You can always leave out an optimization clause if you are not sure about it.
+
+## Parameters<a id="topic1__section4"></a>
+
+ *name*
+The (optionally schema-qualified) name of the operator to be defined. Two operators in the same schema can have the same name if they operate on different data types.
+
+ *funcname*
+The function used to implement this operator (must be an `IMMUTABLE` function).
+
+ *lefttype*
+The data type of the operator's left operand, if any. This option would be omitted for a left-unary operator.
+
+ *righttype*
+The data type of the operator's right operand, if any. This option would be omitted for a right-unary operator.
+
+ *com\_op*
+The optional `COMMUTATOR` clause names an operator that is the commutator of the operator being defined. We say that operator A is the commutator of operator B if (x A y) equals (y B x) for all possible input values x, y. Notice that B is also the commutator of A. For example, operators `<` and `>` for a particular data type are usually each others commutators, and operator + is usually commutative with itself. But operator `-` is usually not commutative with anything. The left operand type of a commutable operator is the same as the right operand type of its commutator, and vice versa. So the name of the commutator operator is all that needs to be provided in the `COMMUTATOR` clause.
+
+ *neg\_op*
+The optional `NEGATOR` clause names an operator that is the negator of the operator being defined. We say that operator A is the negator of operator B if both return Boolean results and (x A y) equals NOT (x B y) for all possible inputs x, y. Notice that B is also the negator of A. For example, `<` and `>=` are a negator pair for most data types. An operator's negator must have the same left and/or right operand types as the operator to be defined, so only the operator name need be given in the `NEGATOR` clause.
+
+ *res\_proc*
+The optional `RESTRICT` names a restriction selectivity estimation function for the operator. Note that this is a function name, not an operator name. `RESTRICT` clauses only make sense for binary operators that return `boolean`. The idea behind a restriction selectivity estimator is to guess what fraction of the rows in a table will satisfy a `WHERE`-clause condition of the form:
+
+``` pre
+column OP constant
+```
+
+for the current operator and a particular constant value. This assists the optimizer by giving it some idea of how many rows will be eliminated by `WHERE` clauses that have this form.
+
+You can usually just use one of the following system standard estimator functions for many of your own operators:
+
+`eqsel` for =
+
+`neqsel` for <>
+
+`scalarltsel` for < or <=
+
+`scalargtsel` for > or >=
+
+ *join\_proc*
+The optional `JOIN` clause names a join selectivity estimation function for the operator. Note that this is a function name, not an operator name. `JOIN` clauses only make sense for binary operators that return `boolean`. The idea behind a join selectivity estimator is to guess what fraction of the rows in a pair of tables will satisfy a `WHERE`-clause condition of the form
+
+``` pre
+table1.column1 OP table2.column2
+```
+
+for the current operator. This helps the optimizer by letting it figure out which of several possible join sequences is likely to take the least work.
+
+You can usually just use one of the following system standard join selectivity estimator functions for many of your own operators:
+
+`eqjoinsel` for =
+
+`neqjoinsel` for <>
+
+`scalarltjoinsel` for < or <=
+
+`scalargtjoinsel` for > or >=
+
+`areajoinsel` for 2D area-based comparisons
+
+`positionjoinsel` for 2D position-based comparisons
+
+`contjoinsel` for 2D containment-based comparisons
+
+HASHES
+The optional `HASHES` clause tells the system that it is permissible to use the hash join method for a join based on this operator. `HASHES` only makes sense for a binary operator that returns `boolean`. The hash join operator can only return true for pairs of left and right values that hash to the same hash code. If two values get put in different hash buckets, the join will never compare them at all, implicitly assuming that the result of the join operator must be false. So it never makes sense to specify `HASHES` for operators that do not represent equality.
+
+To be marked `HASHES`, the join operator must appear in a hash index operator class. Attempts to use the operator in hash joins will fail at run time if no such operator class exists. The system needs the operator class to find the data-type-specific hash function for the operator's input data type. You must also supply a suitable hash function before you can create the operator class. Care should be exercised when preparing a hash function, because there are machine-dependent ways in which it might fail to do the right thing.
+
+MERGES
+The `MERGES` clause, if present, tells the system that it is permissible to use the merge-join method for a join based on this operator. `MERGES` only makes sense for a binary operator that returns `boolean`, and in practice the operator must represent equality for some data type or pair of data types.
+
+Merge join is based on the idea of sorting the left- and right-hand tables into order and then scanning them in parallel. So, both data types must be capable of being fully ordered, and the join operator must be one that can only succeed for pairs of values that fall at the same place in the sort order. In practice this means that the join operator must behave like equality. It is possible to merge-join two distinct data types so long as they are logically compatible. For example, the smallint-versus-integer equality operator is merge-joinable. We only need sorting operators that will bring both data types into a logically compatible sequence.
+
+Execution of a merge join requires that the system be able to identify four operators related to the merge-join equality operator: less-than comparison for the left operand data type, less-than comparison for the right operand data type, less-than comparison between the two data types, and greater-than comparison between the two data types. It is possible to specify these operators individually by name, as the `SORT1`, `SORT2`, `LTCMP`, and `GTCMP` options respectively. The system will fill in the default names if any of these are omitted when `MERGES` is specified.
+
+ *left\_sort\_op*
+If this operator can support a merge join, the less-than operator that sorts the left-hand data type of this operator. `<` is the default if not specified.
+
+ *right\_sort\_op*
+If this operator can support a merge join, the less-than operator that sorts the right-hand data type of this operator. `<` is the default if not specified.
+
+ *less\_than\_op*
+If this operator can support a merge join, the less-than operator that compares the input data types of this operator. `<` is the default if not specified.
+
+ *greater\_than\_op*
+If this operator can support a merge join, the greater-than operator that compares the input data types of this operator. `>` is the default if not specified.
+
+To give a schema-qualified operator name in optional arguments, use the `OPERATOR()` syntax, for example:
+
+``` pre
+COMMUTATOR = OPERATOR(myschema.===) ,
+```
+
+## Notes<a id="topic1__section5"></a>
+
+Any functions used to implement the operator must be defined as `IMMUTABLE`.
+
+## Examples<a id="topic1__section6"></a>
+
+Here is an example of creating an operator for adding two complex numbers, assuming we have already created the definition of type `complex`. First define the function that does the work, then define the operator:
+
+``` pre
+CREATE FUNCTION complex_add(complex, complex)
+ RETURNS complex
+ AS 'filename', 'complex_add'
+ LANGUAGE C IMMUTABLE STRICT;
+CREATE OPERATOR + (
+ leftarg = complex,
+ rightarg = complex,
+ procedure = complex_add,
+ commutator = +
+);
+```
+
+To use this operator in a query:
+
+``` pre
+SELECT (a + b) AS c FROM test_complex;
+```
+
+## Compatibility<a id="topic1__section7"></a>
+
+`CREATE OPERATOR` is a HAWQ language extension. The SQL standard does not provide for user-defined operators.
+
+## See Also<a id="topic1__section8"></a>
+
+[CREATE FUNCTION](CREATE-FUNCTION.html), [CREATE TYPE](CREATE-TYPE.html), [ALTER OPERATOR](ALTER-OPERATOR.html), [DROP OPERATOR](DROP-OPERATOR.html)
diff --git a/reference/sql/CREATE-RESOURCE-QUEUE.html.md.erb b/reference/sql/CREATE-RESOURCE-QUEUE.html.md.erb
new file mode 100644
index 0000000..a37fe5f
--- /dev/null
+++ b/reference/sql/CREATE-RESOURCE-QUEUE.html.md.erb
@@ -0,0 +1,138 @@
+---
+title: CREATE RESOURCE QUEUE
+---
+
+Defines a new resource queue.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+CREATE RESOURCE QUEUE name WITH (queue_attribute=value [, ... ])
+```
+
+where *queue\_attribute* is:
+
+``` pre
+PARENT='queue_name'
+ [ACTIVE_STATEMENTS=integer]
+ MEMORY_LIMIT_CLUSTER=percentage
+ CORE_LIMIT_CLUSTER=percentage
+ [ALLOCATION_POLICY='even']
+ [VSEGMENT_RESOURCE_QUOTA='mem:memory_units']
+ [RESOURCE_OVERCOMMIT_FACTOR=double]
+ [NVSEG_UPPER_LIMIT=integer]
+ [NVSEG_LOWER_LIMIT=integer]
+ [NVSEG_UPPER_LIMIT_PERSEG=double]
+ [NVSEG_LOWER_LIMIT_PERSEG=double]
+
+ memory_units ::= {128mb|256mb|512mb|1024mb|2048mb|4096mb|
+ 8192mb|16384mb|1gb|2gb|4gb|8gb|16gb}
+ percentage ::= integer %
+```
+
+## Description<a id="topic1__section3"></a>
+
+Creates a new resource queue for HAWQ workload management. A resource queue must specify a parent queue. Only a superuser can create a resource queue.
+
+Resource queues with an `ACTIVE_STATEMENTS` threshold set a maximum limit on the number of queries that can be executed by roles assigned to that queue. It controls the number of active queries that are allowed to run at the same time. The value for `ACTIVE_STATEMENTS` should be an integer greater than 0. If not specified, the default value is 20.
+
+When creating the resource queue, use the MEMORY\_LIMIT\_CLUSTER and CORE\_LIMIT\_CLUSTER queue attributes to tune the allowed resource usage of the resource queue. MEMORY\_LIMIT\_CLUSTER and CORE\_LIMIT\_CLUSTER must set to the same value for a resource queue. In addition the sum of the percentages of MEMORY\_LIMIT\_CLUSTER (and CORE\_LIMIT\_CLUSTER) for resource queues that share the same parent cannot exceed 100%.
+
+You can optionally configure the maximum or minimum number of virtual segments to use when executing a query by setting NVSEG\_UPPER\_LIMIT/NVSEG\_LOWER\_LIMIT or NVSEG\_UPPER\_LIMIT\_PERSEG/NVSEG\_LOWER\_LIMIT\_PERSEG attributes for the resource queue.
+
+After defining a resource queue, you can assign a role to the queue by using the [ALTER ROLE](ALTER-ROLE.html) or [CREATE ROLE](CREATE-ROLE.html) command. You can only assign roles to the leaf-level resource queues (resource queues that do not have any children.)
+
+See also [Best Practices for Using Resource Queues](../../bestpractices/managing_resources_bestpractices.html#topic_hvd_pls_wv).
+
+## Parameters<a id="topic1__section4"></a>
+
+ *name*
+Required. The name of the resource queue. The name must not already be in use and must not be `pg_default` or `pg_root`.
+
+PARENT *name*
+Required. The parent queue of the new resource queue. The parent queue must already exist. This attribute is used to organize resource queues into a tree structure. You cannot specify `pg_default` as a parent queue. Resource queues that are parents to other resource queues are also called branch queues. Resource queues without any children are also called leaf queues. If you do not have any existing resource queues, use `pg_root` as the starting point for new resource queues.
+
+The parent queue cannot have any roles assigned.
+
+ACTIVE\_STATEMENTS *integer*
+Optional. Defines the limit of the number of parallel active statements in one leaf queue. The maximum number of connections cannot exceed this limit. If this limit is reached, the HAWQ resource manager queues more query allocation requests. Note that a single session can have several concurrent statement executions that occupy multiple connection resources. The value for `ACTIVE_STATEMENTS` should be an integer greater than 0. The default value is 20.
+
+MEMORY\_LIMIT\_CLUSTER *percentage*
+Required. Defines how much memory a resource queue can consume from its parent resource queue and consequently dispatch to the execution of parallel statements. Since a resource queue obtains its memory from its parent, the actual memory limit is based from its parent queue. The valid values are 1% to 100%. The value of MEMORY\_ LIMIT\_CLUSTER must be identical to the value of CORE\_LIMIT\_CLUSTER. The sum of values for MEMORY\_LIMIT\_CLUSTER of this queue plus other queues that share the same parent cannot exceed 100%. The HAWQ resource manager periodically validates this restriction.
+
+CORE\_LIMIT\_CLUSTER *percentage*
+Required. The percentage of consumable CPU (virtual core) resources that the resource queue can take from its parent resource queue. The valid values are 1% to 100%. The value of MEMORY\_ LIMIT\_CLUSTER must be identical to the value of CORE\_LIMIT\_CLUSTER. The sum of values for CORE\_LIMIT\_CLUSTER of this queue and queues that share the same parent cannot exceed 100%.
+
+ALLOCATION\_POLICY *string*
+Optional. Defines the resource allocation policy for parallel statement execution. The default value is `even`.
+
+**Note:** This release only supports an `even` allocation policy. Even if you do not specify this attribute, the resource queue still applies an `even` allocation policy. Future releases will support alternative allocation policies.
+
+Setting the allocation policy to `even` means resources are always evenly dispatched based on current concurrency. When multiple query resource allocation requests are queued, the resource queue tries to evenly dispatch resources to queued requests until one of the following conditions are encountered:
+
+- There are no more allocated resources in this queue to dispatch, or
+- The ACTIVE\_STATEMENTS limit has been reached
+
+For each query resource allocation request, the HAWQ resource manager determines the minimum and maximum size of a virtual segment based on multiple factors including query cost, user configuration, table properties, and so on. For example, a hash distributed table requires fixed size of virtual segments. With an even allocation policy, the HAWQ resource manager uses the minimum virtual segment size requirement and evenly dispatches resources to each query resource allocation request in the resource queue.
+
+VSEG\_RESOURCE\_QUOTA 'mem:{128mb | 256mb | 512mb | 1024mb | 2048mb | 4096mb | 8192mb | 16384mb | 1gb | 2gb | 4gb | 8gb | 16gb}'
+Optional. This quota defines how resources are split across multiple virtual segments. For example, when the HAWQ resource manager determines that 256GB memory and 128 vcores should be allocated to the current resource queue, there are multiple solutions on how to divide the resources across virtual segments. For example, you could use a) 2GB/1 vcore \* 128 virtual segments or b) 1GB/0.5 vcore \* 256 virtual segments. Therefore, you can use this attribute to make the HAWQ resource manager calculate the number of virtual segments based on how to divide the memory. For example, if `VSEGMENT_RESOURCE_QUOTA``='mem:512mb'`, then the resource queue will use 512MB/0.25 vcore \* 512 virtual segments. The default value is '`mem:256mb`'.
+
+**Note:** To avoid resource fragmentation, make sure that the segment resource capacity configured for HAWQ (in HAWQ Standalone mode: `hawq_rm_memory_limit_perseg`; in YARN mode: `yarn.nodemanager.resource.memory-mb` must be a multiple of the resource quotas for all virtual segments and CPU to memory ratio must be a multiple of the amount configured for `yarn.scheduler.minimum-allocation-mb`.
+
+RESOURCE\_OVERCOMMIT\_FACTOR *double*
+Optional. This factor defines how much a resource can be overcommitted. For example, if RESOURCE\_OVERCOMMIT\_FACTOR is set to 3.0 and MEMORY\_LIMIT\_CLUSTER is set to 30%, then the maximum possible resource allocation in this queue is 90% (30% x 3.0). If the resulting maximum is bigger than 100%, then 100% is adopted. The minimum value that this attribute can be set to is `1.0`. The default value is `2.0`.
+
+NVSEG\_UPPER\_LIMIT *integer* / NVSEG\_UPPER\_LIMIT\_PERSEG *double*
+Optional. These limits restrict the range of number of virtual segments allocated in this resource queue for executing one query statement. NVSEG\_UPPER\_LIMIT defines an upper limit of virtual segments for one statement execution regardless of actual cluster size, while NVSEG\_UPPER\_LIMIT\_PERSEG defines the same limit by using the average number of virtual segments in one physical segment. Therefore, the limit defined by NVSEG\_UPPER\_LIMIT\_PERSEG varies dynamically according to the changing size of the HAWQ cluster.
+
+For example, if you set `NVSEG_UPPER_LIMIT=10` all query resource requests are strictly allocated no more than 10 virtual segments. If you set NVSEG\_UPPER\_LIMIT\_PERSEG=2 and assume that currently there are 5 available HAWQ segments in the cluster, query resource requests are allocated 10 virtual segments at the most.
+
+NVSEG\_UPPER\_LIMIT cannot be set to a lower value than NVSEG\_LOWER\_LIMIT if both limits are enabled. In addition, the upper limit cannot be set to a value larger than the value set in global configuration parameter `hawq_rm_nvseg_perquery_limit` and `hawq_rm_nvseg_perquery_perseg_limit`.
+
+By default, both limits are set to **-1**, which means the limits are disabled. `NVSEG_UPPER_LIMIT` has higher priority than `NVSEG_UPPER_LIMIT_PERSEG`. If both limits are set, then `NVSEG_UPPER_LIMIT_PERSEG` is ignored. If you have enabled resource quotas for the query statement, then these limits are ignored.
+
+**Note:** If the actual lower limit of the number of virtual segments becomes greater than the upper limit, then the lower limit is automatically reduced to be equal to the upper limit. This situation is possible when user sets both `NVSEG_UPPER_LIMIT `and `NVSEG_LOWER_LIMIT_PERSEG`. After expanding the cluster, the dynamic lower limit may become greater than the value set for the fixed upper limit.
+
+NVSEG\_LOWER\_LIMIT *integer* / NVSEG\_LOWER\_LIMIT\_PERSEG *double*
+Optional. These limits specify the minimum number of virtual segments allocated for one statement execution in order to guarantee query performance. NVSEG\_LOWER\_LIMIT defines the lower limit of virtual segments for one statement execution regardless the actual cluster size, while NVSEG\_LOWER\_LIMIT\_PERSEG defines the same limit by the average virtual segment number in one segment. Therefore, the limit defined by NVSEG\_LOWER\_LIMIT\_PERSEG varies dynamically along with the size of HAWQ cluster.
+
+NVSEG\_UPPER\_LIMIT\_PERSEG cannot be less than NVSEG\_LOWER\_LIMIT\_PERSEG if both limits are set enabled.
+
+For example, if you set NVSEG\_LOWER\_LIMIT=10, and one statement execution potentially needs no fewer than 10 virtual segments, then this request has at least 10 virtual segments allocated. If you set NVSEG\_UPPER\_LIMIT\_PERSEG=2, assuming there are currently 5 available HAWQ segments in the cluster, and one statement execution potentially needs no fewer than 10 virtual segments, then the query resource request will be allocated at least 10 virtual segments. If one statement execution needs at most 4 virtual segments, the resource manager will allocate at most 4 virtual segments instead of 10 since this resource request does not need more than 9 virtual segments.
+
+By default, both limits are set to **-1**, which means the limits are disabled. `NVSEG_LOWER_LIMIT` has higher priority than `NVSEG_LOWER_LIMIT_PERSEG`. If both limits are set, then `NVSEG_LOWER_LIMIT_PERSEG` is ignored. If you have enabled resource quotas for the query statement, then these limits are ignored.
+
+**Note:** If the actual lower limit of the number of virtual segments becomes greater than the upper limit, then the lower limit is automatically reduced to be equal to the upper limit. This situation is possible when user sets both `NVSEG_UPPER_LIMIT `and `NVSEG_LOWER_LIMIT_PERSEG`. After expanding the cluster, the dynamic lower limit may become greater than the value set for the fixed upper limit. .
+
+## Notes<a id="topic1__section5"></a>
+
+To check on the configuration of a resource queue, you can query the `pg_resqueue` catalog table. To see the runtime status of all resource queues, you can use the `pg_resqueue_status`. See [Checking Existing Resource Queues](/20/resourcemgmt/ResourceQueues.html#topic_lqy_gls_zt).
+
+`CREATE RESOURCE QUEUE` cannot be run within a transaction.
+
+To see the status of a resource queue, see [Checking Existing Resource Queues](/20/resourcemgmt/ResourceQueues.html#topic_lqy_gls_zt).
+
+## Examples<a id="topic1__section6"></a>
+
+Create a resource queue as a child of `pg_root` with an active query limit of 20 and memory and core limits of 50%:
+
+``` pre
+CREATE RESOURCE QUEUE myqueue WITH (PARENT='pg_root', ACTIVE_STATEMENTS=20,
+MEMORY_LIMIT_CLUSTER=50%, CORE_LIMIT_CLUSTER=50%);
+```
+
+Create a resource queue as a child of pg\_root with memory and CPU limits and a resource overcommit factor:
+
+``` pre
+CREATE RESOURCE QUEUE test_queue_1 WITH (PARENT='pg_root',
+MEMORY_LIMIT_CLUSTER=50%, CORE_LIMIT_CLUSTER=50%, RESOURCE_OVERCOMMIT_FACTOR=2);
+```
+
+## Compatibility<a id="topic1__section7"></a>
+
+`CREATE RESOURCE QUEUE` is a HAWQ extension. There is no provision for resource queues or workload management in the SQL standard.
+
+## See Also<a id="topic1__section8"></a>
+
+[ALTER RESOURCE QUEUE](ALTER-RESOURCE-QUEUE.html)[ALTER ROLE](ALTER-ROLE.html), [CREATE ROLE](CREATE-ROLE.html), [DROP RESOURCE QUEUE](DROP-RESOURCE-QUEUE.html)
diff --git a/reference/sql/CREATE-ROLE.html.md.erb b/reference/sql/CREATE-ROLE.html.md.erb
new file mode 100644
index 0000000..d99e7a8
--- /dev/null
+++ b/reference/sql/CREATE-ROLE.html.md.erb
@@ -0,0 +1,195 @@
+---
+title: CREATE ROLE
+---
+
+Defines a new database role (user or group).
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+CREATE ROLE name [[WITH] option [ ... ]]
+```
+
+where *option* can be:
+
+``` pre
+ SUPERUSER | NOSUPERUSER
+ | CREATEDB | NOCREATEDB
+ | CREATEROLE | NOCREATEROLE
+ | CREATEEXTTABLE | NOCREATEEXTTABLE
+ [ ( attribute='value'[, ...] ) ]
+ where attributes and value are:
+ type='readable'|'writable'
+ protocol='gpfdist'|'http'
+ | INHERIT | NOINHERIT
+ | LOGIN | NOLOGIN
+ | CONNECTION LIMIT connlimit
+ | [ ENCRYPTED | UNENCRYPTED ] PASSWORD 'password'
+ | VALID UNTIL 'timestamp'
+ | IN ROLE rolename [, ...]
+ | ROLE rolename [, ...]
+ | ADMIN rolename [, ...]
+ | RESOURCE QUEUE queue_name
+ | [ DENY deny_point ]
+ | [ DENY BETWEEN deny_point AND deny_point]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`CREATE ROLE` adds a new role to a HAWQ system. A role is an entity that can own database objects and have database privileges. A role can be considered a user, a group, or both depending on how it is used. You must have `CREATEROLE` privilege or be a database superuser to use this command.
+
+Note that roles are defined at the system-level and are valid for all databases in your HAWQ system.
+
+## Parameters<a id="topic1__section4"></a>
+
+ *name*
+The name of the new role.
+
+SUPERUSER,
+NOSUPERUSER
+If `SUPERUSER` is specified, the role being defined will be a superuser, who can override all access restrictions within the database. Superuser status is dangerous and should be used only when really needed. You must yourself be a superuser to create a new superuser. `NOSUPERUSER` is the default.
+
+CREATEDB,
+NOCREATEDB
+If `CREATEDB` is specified, the role being defined will be allowed to create new databases. `NOCREATEDB` (the default) will deny a role the ability to create databases.
+
+CREATEROLE,
+NOCREATEROLE
+If `CREATEDB` is specified, the role being defined will be allowed to create new roles, alter other roles, and drop other roles. `NOCREATEROLE` (the default) will deny a role the ability to create roles or modify roles other than their own.
+
+CREATEEXTTABLE,
+NOCREATEEXTTABLE
+If `CREATEEXTTABLE` is specified, the role being defined is allowed to create external tables. The default `type` is `readable` and the default `protocol` is `gpfdist` if not specified. `NOCREATEEXTTABLE` (the default) denies the role the ability to create external tables. Using the `file` protocol when creating external tables is not supported. This is because HAWQ cannot guarantee scheduling executors on a specific host. Likewise, you cannot use the `EXECUTE` command with `ON ALL` and `ON HOST` for the same reason. Use the `ON MASTER/number/SEGMENT segment_id` to specify which segment instances are to execute the command.
+
+INHERIT,
+NOINHERIT
+If specified, `INHERIT` (the default) allows the role to use whatever database privileges have been granted to all roles it is directly or indirectly a member of. With `NOINHERIT`, membership in another role only grants the ability to `SET ROLE` to that other role.
+
+LOGIN,
+NOLOGIN
+If specified, `LOGIN` allows a role to log in to a database. A role having the `LOGIN` attribute can be thought of as a user. Roles with `NOLOGIN` (the default) are useful for managing database privileges, and can be thought of as groups.
+
+CONNECTION LIMIT *connlimit*
+The number maximum of concurrent connections this role can make. The default of -1 means there is no limitation.
+
+<!-- -->
+
+PASSWORD *password*
+Sets the user password for roles with the `LOGIN` attribute. If you do not plan to use password authentication you can omit this option. If no password is specified, the password will be set to null and password authentication will always fail for that user. A null password can optionally be written explicitly as `PASSWORD NULL`.
+
+ENCRYPTED,
+UNENCRYPTED
+These key words control whether the password is stored encrypted in the system catalogs. (If neither is specified, the default behavior is determined by the configuration parameter *password\_encryption*.) If the presented password string is already in MD5-encrypted format, then it is stored encrypted as-is, regardless of whether `ENCRYPTED` or `UNENCRYPTED` is specified (since the system cannot decrypt the specified encrypted password string). This allows reloading of encrypted passwords during dump/restore.
+
+Note that older clients may lack support for the MD5 authentication mechanism that is needed to work with passwords that are stored encrypted.
+
+VALID UNTIL '*timestamp*'
+The VALID UNTIL clause sets a date and time after which the role's password is no longer valid. If this clause is omitted the password will never expire.
+
+IN ROLE *rolename*
+Adds the new role as a member of the named roles. Note that there is no option to add the new role as an administrator; use a separate `GRANT` command to do that.
+
+ROLE *rolename*
+Adds the named roles as members of this role, making this new role a group.
+
+ADMIN *rolename*
+The `ADMIN` clause is like `ROLE`, but the named roles are added to the new role `WITH ADMIN OPTION`, giving them the right to grant membership in this role to others.
+
+RESOURCE QUEUE *queue\_name*
+The name of the resource queue to which the new user-level role is to be assigned. Only roles with `LOGIN` privilege can be assigned to a resource queue. The special keyword `NONE` means that the role is assigned to the default resource queue. A role can only belong to one resource queue.
+
+DENY *deny\_point*,
+DENY BETWEEN *deny\_point* AND *deny\_point*
+The `DENY` and `DENY BETWEEN` keywords set time-based constraints that are enforced at login. `DENY` sets a day or a day and time to deny access. `DENY BETWEEN` sets an interval during which access is denied. Both use the parameter *deny\_point* that has the following format:
+
+``` pre
+DAY day [ TIME 'time' ]
+```
+
+The two parts of the `deny_point` parameter use the following formats:
+
+For `day`:
+
+``` pre
+{'Sunday' | 'Monday' | 'Tuesday' |'Wednesday' | 'Thursday' | 'Friday' |
+'Saturday' | 0-6 }
+```
+
+For `time`:
+
+``` pre
+{ 00-23 : 00-59 | 01-12 : 00-59 { AM | PM }}
+```
+
+The `DENY BETWEEN` clause uses two *deny\_point* parameters:
+
+``` pre
+DENY BETWEEN deny_point AND deny_point
+
+```
+
+## Notes<a id="topic1__section5"></a>
+
+The preferred way to add and remove role members (manage groups) is to use [GRANT](GRANT.html) and [REVOKE](REVOKE.html).
+
+The `VALID UNTIL` clause defines an expiration time for a password only, not for the role. The expiration time is not enforced when logging in using a non-password-based authentication method.
+
+The `INHERIT` attribute governs inheritance of grantable privileges (access privileges for database objects and role memberships). It does not apply to the special role attributes set by `CREATE ROLE` and `ALTER ROLE`. For example, being a member of a role with `CREATEDB` privilege does not immediately grant the ability to create databases, even if `INHERIT` is set.
+
+The `INHERIT` attribute is the default for reasons of backwards compatibility. In prior releases of HAWQ, users always had access to all privileges of groups they were members of. However, `NOINHERIT` provides a closer match to the semantics specified in the SQL standard.
+
+Be careful with the `CREATEROLE` privilege. There is no concept of inheritance for the privileges of a `CREATEROLE`-role. That means that even if a role does not have a certain privilege but is allowed to create other roles, it can easily create another role with different privileges than its own (except for creating roles with superuser privileges). For example, if a role has the `CREATEROLE` privilege but not the `CREATEDB` privilege, it can create a new role with the `CREATEDB` privilege. Therefore, regard roles that have the `CREATEROLE` privilege as almost-superuser-roles.
+
+The `CONNECTION LIMIT` option is never enforced for superusers.
+
+Caution must be exercised when specifying an unencrypted password with this command. The password will be transmitted to the server in clear-text, and it might also be logged in the client's command history or the server log. The client program `createuser`, however, transmits the password encrypted. Also, psql contains a command `\password` that can be used to safely change the password later.
+
+## Examples<a id="topic1__section6"></a>
+
+Create a role that can log in, but don't give it a password:
+
+``` pre
+CREATE ROLE jonathan LOGIN;
+```
+
+Create a role that belongs to a resource queue:
+
+``` pre
+CREATE ROLE jonathan LOGIN RESOURCE QUEUE poweruser;
+```
+
+Create a role with a password that is valid until the end of 2009 (`CREATE USER` is the same as `CREATE ROLE` except that it implies `LOGIN`):
+
+``` pre
+CREATE USER joelle WITH PASSWORD 'jw8s0F4' VALID UNTIL '2010-01-01';
+```
+
+Create a role that can create databases and manage other roles:
+
+``` pre
+CREATE ROLE admin WITH CREATEDB CREATEROLE;
+```
+
+Create a role that does not allow login access on Sundays:
+
+``` pre
+CREATE ROLE user3 DENY DAY 'Sunday';
+```
+
+## Compatibility<a id="topic1__section7"></a>
+
+The SQL standard defines the concepts of users and roles, but it regards them as distinct concepts and leaves all commands defining users to be specified by the database implementation. In HAWQ, users and roles are unified into a single type of object. Roles therefore have many more optional attributes than they do in the standard.
+
+`CREATE ROLE` is in the SQL standard, but the standard only requires the syntax:
+
+``` pre
+CREATE ROLE name [WITH ADMIN rolename]
+```
+
+Allowing multiple initial administrators, and all the other options of `CREATE ROLE`, are HAWQ extensions.
+
+The behavior specified by the SQL standard is most closely approximated by giving users the `NOINHERIT` attribute, while roles are given the `INHERIT` attribute.
+
+## See Also<a id="topic1__section8"></a>
+
+[SET ROLE](SET-ROLE.html), [ALTER ROLE](ALTER-ROLE.html), [DROP ROLE](DROP-ROLE.html), [GRANT](GRANT.html), [REVOKE](REVOKE.html), [CREATE RESOURCE QUEUE](CREATE-RESOURCE-QUEUE.html)
diff --git a/reference/sql/CREATE-SCHEMA.html.md.erb b/reference/sql/CREATE-SCHEMA.html.md.erb
new file mode 100644
index 0000000..1432b71
--- /dev/null
+++ b/reference/sql/CREATE-SCHEMA.html.md.erb
@@ -0,0 +1,63 @@
+---
+title: CREATE SCHEMA
+---
+
+Defines a new schema.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+CREATE SCHEMA schema_name [AUTHORIZATION username]
+ [schema_element [ ... ]]
+
+CREATE SCHEMA AUTHORIZATION rolename [schema_element [ ... ]]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`CREATE SCHEMA` enters a new schema into the current database. The schema name must be distinct from the name of any existing schema in the current database.
+
+A schema is essentially a namespace: it contains named objects (tables, data types, functions, and operators) whose names may duplicate those of other objects existing in other schemas. Named objects are accessed either by qualifying their names with the schema name as a prefix, or by setting a search path that includes the desired schema(s). A `CREATE` command specifying an unqualified object name creates the object in the current schema (the one at the front of the search path, which can be determined with the function `current_schema`).
+
+Optionally, `CREATE SCHEMA` can include subcommands to create objects within the new schema. The subcommands are treated essentially the same as separate commands issued after creating the schema, except that if the `AUTHORIZATION` clause is used, all the created objects will be owned by that role.
+
+## Parameters<a id="topic1__section4"></a>
+
+ *schema\_name*
+The name of a schema to be created. If this is omitted, the user name is used as the schema name. The name cannot begin with `pg_`, as such names are reserved for system catalog schemas.
+
+ *rolename*
+The name of the role who will own the schema. If omitted, defaults to the role executing the command. Only superusers may create schemas owned by roles other than themselves.
+
+ *schema\_element*
+An SQL statement defining an object to be created within the schema. Currently, only `CREATE TABLE`, `CREATE VIEW`, `CREATE INDEX`, `CREATE SEQUENCE`, and `GRANT` are accepted as clauses within `CREATE SCHEMA`. Other kinds of objects may be created in separate commands after the schema is created.
+
+## Notes<a id="topic1__section5"></a>
+
+To create a schema, the invoking user must have the `CREATE` privilege for the current database or be a superuser.
+
+## Examples<a id="topic1__section6"></a>
+
+Create a schema:
+
+``` pre
+CREATE SCHEMA myschema;
+```
+
+Create a schema for role `joe` (the schema will also be named `joe`):
+
+``` pre
+CREATE SCHEMA AUTHORIZATION joe;
+```
+
+## Compatibility<a id="topic1__section7"></a>
+
+The SQL standard allows a `DEFAULT CHARACTER SET` clause in `CREATE SCHEMA`, as well as more subcommand types than are presently accepted by HAWQ.
+
+The SQL standard specifies that the subcommands in `CREATE SCHEMA` may appear in any order. The present HAWQ implementation does not handle all cases of forward references in subcommands; it may sometimes be necessary to reorder the subcommands in order to avoid forward references.
+
+According to the SQL standard, the owner of a schema always owns all objects within it. HAWQ allows schemas to contain objects owned by users other than the schema owner. This can happen only if the schema owner grants the `CREATE` privilege on the schema to someone else.
+
+## See Also<a id="topic1__section8"></a>
+
+[DROP SCHEMA](DROP-SCHEMA.html)
diff --git a/reference/sql/CREATE-SEQUENCE.html.md.erb b/reference/sql/CREATE-SEQUENCE.html.md.erb
new file mode 100644
index 0000000..96681b5
--- /dev/null
+++ b/reference/sql/CREATE-SEQUENCE.html.md.erb
@@ -0,0 +1,135 @@
+---
+title: CREATE SEQUENCE
+---
+
+Defines a new sequence generator.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+CREATE [TEMPORARY | TEMP] SEQUENCE name
+ [INCREMENT [BY] value]
+ [MINVALUE minvalue | NO MINVALUE]
+ [MAXVALUE maxvalue | NO MAXVALUE]
+ [START [ WITH ] start]
+ [CACHE cache]
+ [[NO] CYCLE]
+ [OWNED BY { table.column | NONE }]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`CREATE SEQUENCE` creates a new sequence number generator. This involves creating and initializing a new special single-row table. The generator will be owned by the user issuing the command.
+
+If a schema name is given, then the sequence is created in the specified schema. Otherwise it is created in the current schema. Temporary sequences exist in a special schema, so a schema name may not be given when creating a temporary sequence. The sequence name must be distinct from the name of any other sequence, table, or view in the same schema.
+
+After a sequence is created, you use the `nextval` function to operate on the sequence. For example, to insert a row into a table that gets the next value of a sequence:
+
+``` pre
+INSERT INTO distributors VALUES (nextval('myserial'), 'acme');
+```
+
+You can also use the function `setval` to operate on a sequence, but only for queries that do not operate on distributed data. For example, the following query is allowed because it resets the sequence counter value for the sequence generator process on the master:
+
+``` pre
+SELECT setval('myserial', 201);
+```
+
+But the following query will be rejected in HAWQ because it operates on distributed data:
+
+``` pre
+INSERT INTO product VALUES (setval('myserial', 201), 'gizmo');
+```
+
+In a regular (non-distributed) database, functions that operate on the sequence go to the local sequence table to get values as they are needed. In HAWQ, however, keep in mind that each segment is its own distinct database process. Therefore the segments need a single point of truth to go for sequence values so that all segments get incremented correctly and the sequence moves forward in the right order. A sequence server process runs on the master and is the point-of-truth for a sequence in a HAWQ distributed database. Segments get sequence values at runtime from the master.
+
+Because of this distributed sequence design, there are some limitations on the functions that operate on a sequence in HAWQ:
+
+- `lastval` and `currval` functions are not supported.
+- `setval` can only be used to set the value of the sequence generator on the master, it cannot be used in subqueries to update records on distributed table data.
+- `nextval` sometimes grabs a block of values from the master for a segment to use, depending on the query. So values may sometimes be skipped in the sequence if all of the block turns out not to be needed at the segment level. Note that a regular PostgreSQL database does this too, so this is not something unique to HAWQ.
+
+Although you cannot update a sequence directly, you can use a query like:
+
+``` pre
+SELECT * FROM sequence_name;
+```
+
+to examine the parameters and current state of a sequence. In particular, the *last\_value* field of the sequence shows the last value allocated by any session.
+
+## Parameters<a id="topic1__section4"></a>
+
+TEMPORARY | TEMP
+If specified, the sequence object is created only for this session, and is automatically dropped on session exit. Existing permanent sequences with the same name are not visible (in this session) while the temporary sequence exists, unless they are referenced with schema-qualified names.
+
+ *name*
+The name (optionally schema-qualified) of the sequence to be created.
+
+ *increment*
+Specifies which value is added to the current sequence value to create a new value. A positive value will make an ascending sequence, a negative one a descending sequence. The default value is 1.
+
+ *minvalue*
+NO MINVALUE
+Determines the minimum value a sequence can generate. If this clause is not supplied or `NO MINVALUE` is specified, then defaults will be used. The defaults are 1 and -263-1 for ascending and descending sequences, respectively.
+
+ *maxvalue*
+NO MAXVALUE
+Determines the maximum value for the sequence. If this clause is not supplied or `NO MAXVALUE` is specified, then default values will be used. The defaults are 263-1 and -1 for ascending and descending sequences, respectively.
+
+ *start*
+Allows the sequence to begin anywhere. The default starting value is *minvalue* for ascending sequences and *maxvalue* for descending ones.
+
+ *cache*
+Specifies how many sequence numbers are to be preallocated and stored in memory for faster access. The minimum (and default) value is 1 (no cache).
+
+CYCLE
+NO CYCLE
+Allows the sequence to wrap around when the *maxvalue* (for ascending) or *minvalue* (for descending) has been reached. If the limit is reached, the next number generated will be the *minvalue* (for ascending) or *maxvalue* (for descending). If `NO CYCLE` is specified, any calls to `nextval` after the sequence has reached its maximum value will return an error. If not specified, `NO CYCLE` is the default.
+
+OWNED BY *table.column*
+OWNED BY NONE
+Causes the sequence to be associated with a specific table column, such that if that column (or its whole table) is dropped, the sequence will be automatically dropped as well. The specified table must have the same owner and be in the same schema as the sequence. `OWNED BY NONE`, the default, specifies that there is no such association.
+
+## Notes<a id="topic1__section5"></a>
+
+Sequences are based on bigint arithmetic, so the range cannot exceed the range of an eight-byte integer (-9223372036854775808 to 9223372036854775807).
+
+Although multiple sessions are guaranteed to allocate distinct sequence values, the values may be generated out of sequence when all the sessions are considered. For example, session A might reserve values 1..10 and return `nextval=1`, then session B might reserve values 11..20 and return `nextval=11` before session A has generated nextval=2. Thus, you should only assume that the `nextval` values are all distinct, not that they are generated purely sequentially. Also, *last\_value* will reflect the latest value reserved by any session, whether or not it has yet been returned by `nextval`.
+
+## Examples<a id="topic1__section6"></a>
+
+Create a sequence named *myseq*:
+
+``` pre
+CREATE SEQUENCE myseq START 101;
+```
+
+Insert a row into a table that gets the next value:
+
+``` pre
+INSERT INTO distributors VALUES (nextval('myseq'), 'acme');
+```
+
+Reset the sequence counter value on the master:
+
+``` pre
+SELECT setval('myseq', 201);
+```
+
+Illegal use of `setval` in HAWQ (setting sequence values on distributed data):
+
+``` pre
+INSERT INTO product VALUES (setval('myseq', 201), 'gizmo');
+```
+
+## Compatibility<a id="topic1__section7"></a>
+
+`CREATE SEQUENCE` conforms to the SQL standard, with the following exceptions:
+
+- The `AS data_type ` expression specified in the SQL standard is not supported.
+- Obtaining the next value is done using the `nextval()` function instead of the `NEXT VALUE FOR` expression specified in the SQL standard.
+- The `OWNED BY` clause is a HAWQ extension.
+
+## See Also<a id="topic1__section8"></a>
+
+[DROP SEQUENCE](DROP-SEQUENCE.html)
diff --git a/reference/sql/CREATE-TABLE-AS.html.md.erb b/reference/sql/CREATE-TABLE-AS.html.md.erb
new file mode 100644
index 0000000..fa189ca
--- /dev/null
+++ b/reference/sql/CREATE-TABLE-AS.html.md.erb
@@ -0,0 +1,126 @@
+---
+title: CREATE TABLE AS
+---
+
+Defines a new table from the results of a query.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+CREATE [ [GLOBAL | LOCAL] {TEMPORARY | TEMP} ] TABLE table_name
+ [(column_name [, ...] )]
+ [ WITH ( storage_parameter=value [, ... ] )
+ [ON COMMIT {PRESERVE ROWS | DELETE ROWS | DROP}]
+ [TABLESPACE tablespace]
+ AS query
+ [DISTRIBUTED BY (column, [ ... ] ) | DISTRIBUTED RANDOMLY]
+```
+
+where *storage\_parameter* is:
+
+``` pre
+ APPENDONLY={TRUE}
+ BLOCKSIZE={8192-2097152}
+ bucketnum={x}
+ ORIENTATION={ROW | PARQUET}
+ COMPRESSTYPE={ZLIB | SNAPPY | GZIP | NONE}
+ COMPRESSLEVEL={0-9 | 1}
+ FILLFACTOR={10-100}
+ OIDS=[TRUE | FALSE]
+ PAGESIZE={1024-1073741823}
+ ROWGROUPSIZE={1024-1073741823}
+```
+
+## Description<a id="topic1__section3"></a>
+
+`CREATE TABLE AS` creates a table and fills it with data computed by a [SELECT](SELECT.html) command. The table columns have the names and data types associated with the output columns of the `SELECT`, however you can override the column names by giving an explicit list of new column names.
+
+`CREATE TABLE AS` creates a new table and evaluates the query just once to fill the new table initially. The new table will not track subsequent changes to the source tables of the query.
+
+## Parameters<a id="topic1__section4"></a>
+
+GLOBAL | LOCAL
+These keywords are present for SQL standard compatibility, but have no effect in HAWQ.
+
+TEMPORARY | TEMP
+If specified, the new table is created as a temporary table. Temporary tables are automatically dropped at the end of a session, or optionally at the end of the current transaction (see `ON COMMIT`). Existing permanent tables with the same name are not visible to the current session while the temporary table exists, unless they are referenced with schema-qualified names. Any indexes created on a temporary table are automatically temporary as well.
+
+ *table\_name*
+The name (optionally schema-qualified) of the new table to be created.
+
+ *column\_name*
+The name of a column in the new table. If column names are not provided, they are taken from the output column names of the query. If the table is created from an `EXECUTE` command, a column name list cannot be specified.
+
+WITH ( *storage\_parameter=value* )
+The `WITH` clause can be used to set storage options for the table or its indexes. Note that you can also set different storage parameters on a particular partition or subpartition by declaring the `WITH` clause in the partition specification. The following storage options are available:
+
+**APPENDONLY** — Set to `TRUE` to create the table as an append-only table. If `FALSE`, an error message displays stating that heap tables are not supported.
+
+**BLOCKSIZE** — Set to the size, in bytes for each block in a table. The `BLOCKSIZE` must be between 8192 and 2097152 bytes, and be a multiple of 8192. The default is 32768.
+
+**bucketnum** — Set to the number of hash buckets to be used in creating a hash-distributed table. If changing the number of hash buckets, use `WITH` to specify `bucketnum` in creating a hash-distributed table. If distribution is specified by column, the table will inherit the value.
+
+**ORIENTATION** — Set to `row` (the default) for row-oriented storage, or `parquet`. This option is only valid if `APPENDONLY=TRUE`. Heap-storage tables can only be row-oriented.
+
+**COMPRESSTYPE** — Set to `ZLIB` (the default) to specify the type of compression used. QuickLZ has been deprecated. Zlib compression provides more compact compression ratios at lower speeds. This option is only valid if `APPENDONLY=TRUE`.
+
+**COMPRESSLEVEL** — For zlib compression of append-only tables, set to a value between 1 (fastest compression) to 9 (highest compression ratio). If not declared, the default is 1. This option is only valid if `APPENDONLY=TRUE`.
+
+**OIDS** — Set to `OIDS=FALSE` (the default) so that rows do not have object identifiers assigned to them. Do not enable OIDS when creating a table. On large tables, such as those in a typical HAWQ system, using OIDs for table rows can cause wrap-around of the 32-bit OID counter. Once the counter wraps around, OIDs can no longer be assumed to be unique, which not only makes them useless to user applications, but can also cause problems in the HAWQ system catalog tables. In addition, excluding OIDs from a table reduces the space required to store the table on disk by 4 bytes per row, slightly improving performance.
+
+ON COMMIT
+The behavior of temporary tables at the end of a transaction block can be controlled using `ON COMMIT`. The three options are:
+
+**PRESERVE ROWS** — No special action is taken at the ends of transactions for temporary tables. This is the default behavior.
+
+**DELETE ROWS** — All rows in the temporary table will be deleted at the end of each transaction block. Essentially, an automatic `TRUNCATE` is done at each commit.
+
+**DROP** — The temporary table will be dropped at the end of the current transaction block.
+
+TABLESPACE *tablespace*
+The tablespace is the name of the tablespace in which the new table is to be created. If not specified, the database's default tablespace is used.
+
+AS *query*
+A [SELECT](SELECT.html) command, or an [EXECUTE](EXECUTE.html) command that runs a prepared `SELECT` query.
+
+DISTRIBUTED BY (*column*, \[ ... \] )
+DISTRIBUTED RANDOMLY
+Used to declare the HAWQ distribution policy for the table. The default is RANDOM distribution. `DISTIBUTED BY` can use hash distribution with one or more columns declared as the distribution key. If hash distribution is desired, it can be specified using `bucketnum` attribute, using the first eligible column of the table as the distribution key.
+
+## Notes<a id="topic1__section5"></a>
+
+This command is functionally similar to [SELECT INTO](SELECT-INTO.html), but it is preferred since it is less likely to be confused with other uses of the `SELECT INTO` syntax. Furthermore, `CREATE TABLE AS` offers a superset of the functionality offered by `SELECT INTO`.
+
+`CREATE TABLE AS` can be used for fast data loading from external table data sources. See [CREATE EXTERNAL TABLE](CREATE-EXTERNAL-TABLE.html).
+
+## Examples<a id="topic1__section6"></a>
+
+Create a new table `films_recent` consisting of only recent entries from the table `films`:
+
+``` pre
+CREATE TABLE films_recent AS SELECT * FROM films WHERE
+date_prod >= '2007-01-01';
+```
+
+Create a new temporary table `films_recent`, consisting of only recent entries from the table films, using a prepared statement. The new table has OIDs and will be dropped at commit:
+
+``` pre
+PREPARE recentfilms(date) AS SELECT * FROM films WHERE
+date_prod > $1;
+CREATE TEMP TABLE films_recent WITH (OIDS) ON COMMIT DROP AS
+EXECUTE recentfilms('2007-01-01');
+```
+
+## Compatibility<a id="topic1__section7"></a>
+
+`CREATE TABLE AS` conforms to the SQL standard, with the following exceptions:
+
+- The standard requires parentheses around the subquery clause; in HAWQ, these parentheses are optional.
+- The standard defines a `WITH [NO] DATA` clause; this is not currently implemented by HAWQ. The behavior provided by HAWQ is equivalent to the standard's `WITH DATA` case. `WITH NO DATA` can be simulated by appending `LIMIT 0` to the query.
+- HAWQ handles temporary tables differently from the standard; see `CREATE TABLE` for details.
+- The `WITH` clause is a HAWQ extension; neither storage parameters nor `OIDs` are in the standard.
+- The HAWQ concept of tablespaces is not part of the standard. The `TABLESPACE` clause is an extension.
+
+## See Also<a id="topic1__section8"></a>
+
+[CREATE EXTERNAL TABLE](CREATE-EXTERNAL-TABLE.html), [EXECUTE](EXECUTE.html), [SELECT](SELECT.html), [SELECT INTO](SELECT-INTO.html)
diff --git a/reference/sql/CREATE-TABLE.html.md.erb b/reference/sql/CREATE-TABLE.html.md.erb
new file mode 100644
index 0000000..716154c
--- /dev/null
+++ b/reference/sql/CREATE-TABLE.html.md.erb
@@ -0,0 +1,449 @@
+---
+title: CREATE TABLE
+---
+
+Defines a new table.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+CREATE [[GLOBAL | LOCAL] {TEMPORARY | TEMP}] TABLE table_name (
+[ { column_name data_type [ DEFAULT default_expr ]
+ [column_constraint [ ... ]
+[ ENCODING ( storage_directive [,...] ) ]
+]
+ | table_constraint
+ | LIKE other_table [{INCLUDING | EXCLUDING}
+ {DEFAULTS | CONSTRAINTS}] ...} ]
+ [, ... ] ]
+ [column_reference_storage_directive [, …] ]
+ )
+ [ INHERITS ( parent_table [, ... ] ) ]
+ [ WITH ( storage_parameter=value [, ... ] )
+ [ ON COMMIT {PRESERVE ROWS | DELETE ROWS | DROP} ]
+ [ TABLESPACE tablespace ]
+ [ DISTRIBUTED BY (column, [ ... ] ) | DISTRIBUTED RANDOMLY ]
+ [ PARTITION BY partition_type (column)
+ [ SUBPARTITION BY partition_type (column) ]
+ [ SUBPARTITION TEMPLATE ( template_spec ) ]
+ [...]
+ ( partition_spec )
+ | [ SUBPARTITION BY partition_type (column) ]
+ [...]
+ ( partition_spec
+ [ ( subpartition_spec
+ [(...)]
+ ) ]
+ )
+```
+
+where *column\_constraint* is:
+
+``` pre
+ [CONSTRAINT constraint_name]
+ NOT NULL | NULL
+ | CHECK ( expression )
+```
+
+where *storage\_directive* for a column is:
+
+``` pre
+ COMPRESSTYPE={ZLIB | SNAPPY | GZIP | NONE}
+ | COMPRESSLEVEL={0-9}
+ | BLOCKSIZE={8192-2097152}
+```
+
+where *storage\_parameter* for a table is:
+
+``` pre
+ APPENDONLY={TRUE}
+ BLOCKSIZE={8192-2097152}
+ bucketnum={x}
+ ORIENTATION={ROW | PARQUET}
+ COMPRESSTYPE={ZLIB | SNAPPY | GZIP | NONE}
+ COMPRESSLEVEL={0-9}
+ FILLFACTOR={10-100}
+ OIDS=[TRUE|FALSE]
+ PAGESIZE={1024-1073741823}
+ ROWGROUPSIZE={1024-1073741823}
+```
+
+and *table\_constraint* is:
+
+``` pre
+ [CONSTRAINT constraint_name]
+ | CHECK ( expression )
+```
+
+where *partition\_type* is:
+
+``` pre
+ LIST
+ | RANGE
+```
+
+where *partition\_specification* is:
+
+``` pre
+ partition_element [, ...]
+```
+
+and *partition\_element* is:
+
+``` pre
+ DEFAULT PARTITION name
+ | [PARTITION name] VALUES (list_value [,...] )
+ | [PARTITION name]
+ START ([datatype] 'start_value') [INCLUSIVE | EXCLUSIVE]
+ [ END ([datatype] 'end_value') [INCLUSIVE | EXCLUSIVE] ]
+ [ EVERY ([datatype] [number | INTERVAL] 'interval_value') ]
+ | [PARTITION name]
+ END ([datatype] 'end_value') [INCLUSIVE | EXCLUSIVE]
+ [ EVERY ([datatype] [number | INTERVAL] 'interval_value') ]
+[ WITH ( partition_storage_parameter=value [, ... ] ) ]
+[column_reference_storage_directive [, …] ]
+[ TABLESPACE tablespace ]
+```
+
+where *subpartition\_spec* or *template\_spec* is:
+
+``` pre
+ subpartition_element [, ...]
+```
+
+and *subpartition\_element* is:
+
+``` pre
+ DEFAULT SUBPARTITION name
+ | [SUBPARTITION name] VALUES (list_value [,...] )
+ | [SUBPARTITION name]
+ START ([datatype] 'start_value') [INCLUSIVE | EXCLUSIVE]
+ [ END ([datatype] 'end_value') [INCLUSIVE | EXCLUSIVE] ]
+ [ EVERY ([datatype] [number | INTERVAL] 'interval_value') ]
+ | [SUBPARTITION name]
+ END ([datatype] 'end_value') [INCLUSIVE | EXCLUSIVE]
+ [ EVERY ([datatype] [number | INTERVAL] 'interval_value') ]
+[ WITH ( partition_storage_parameter=value [, ... ] ) ]
+[column_reference_storage_directive [, …] ]
+[ TABLESPACE tablespace ]
+```
+
+where *storage\_directive* is:
+
+``` pre
+ COMPRESSTYPE={ZLIB | SNAPPY | GZIP | NONE}
+ | COMPRESSLEVEL={0-9}
+ | BLOCKSIZE={8192-2097152}
+```
+
+where *column\_reference\_storage\_directive* is:
+
+``` pre
+ COLUMN column_name ENCODING (storage_directive [, ... ] ), ...
+ |
+ DEFAULT COLUMN ENCODING (storage_directive [, ... ] )
+```
+
+where *storage\_parameter* for a partition is:
+
+``` pre
+ APPENDONLY={TRUE}
+ BLOCKSIZE={8192-2097152}
+ ORIENTATION={ROW | PARQUET}
+ COMPRESSTYPE={ZLIB | SNAPPY | GZIP | NONE}
+ COMPRESSLEVEL={0-9}
+ FILLFACTOR={10-100}
+ OIDS=[TRUE|FALSE]
+ PAGESIZE={1024-1073741823}
+ ROWGROUPSIZE={1024-1073741823}
+```
+
+## Description<a id="topic1__section3"></a>
+
+`CREATE TABLE` creates a new, initially empty table in the current database. The table is owned by the user issuing the command. If a schema name is given then the table is created in the specified schema. Otherwise it is created in the current schema. Temporary tables exist in a special schema, so a schema name may not be given when creating a temporary table. The name of the table must be distinct from the name of any other table, external table, sequence, or view in the same schema.
+
+The optional constraint clauses specify conditions that new or updated rows must satisfy for an insert or update operation to succeed. A constraint is an SQL object that helps define the set of valid values in the table in various ways. Constraints apply to tables, not to partitions. You cannot add a constraint to a partition or subpartition.
+
+There are two ways to define constraints: table constraints and column constraints. A column constraint is defined as part of a column definition. A table constraint definition is not tied to a particular column, and it can encompass more than one column. Every column constraint can also be written as a table constraint; a column constraint is only a notational convenience for use when the constraint only affects one column.
+
+When creating a table, there is an additional clause to declare the HAWQ distribution policy. If a `DISTRIBUTED BY` clause is not supplied, HAWQ assigns a `RANDOM` distribution policy to the table, where the rows are distributed based on a round-robin or random distribution. You can also choose to distribute data with a hash-based policy, where the `bucketnum` attribute sets the number of hash buckets used by a hash-distributed table. Columns of geometric or user-defined data types are not eligible as HAWQ distribution key columns. The number of buckets affects how many virtual segments will be used in processing.
+
+By default, a HASH distributed table is created with the number of hash buckets specified by the parameter *default\_hash\_table\_bucket\_number*. This can be changed in session level or in the create table DDL with `bucketnum` storage parameter.
+
+**Note:** Column-oriented tables are no longer supported. Use Parquet tables for HAWQ internal tables.
+
+The `PARTITION BY` clause allows you to divide the table into multiple sub-tables (or parts) that, taken together, make up the parent table and share its schema. Though the sub-tables exist as independent tables, HAWQ restricts their use in important ways. Internally, partitioning is implemented as a special form of inheritance. Each child table partition is created with a distinct `CHECK` constraint which limits the data the table can contain, based on some defining criteria. The `CHECK` constraints are also used by the query planner to determine which table partitions to scan in order to satisfy a given query predicate. These partition constraints are managed automatically by HAWQ.
+
+## Parameters<a id="topic1__section4"></a>
+
+GLOBAL | LOCAL
+These keywords are present for SQL standard compatibility, but have no effect in HAWQ.
+
+TEMPORARY | TEMP
+If specified, the table is created as a temporary table. Temporary tables are automatically dropped at the end of a session, or optionally at the end of the current transaction (see `ON COMMIT`). Existing permanent tables with the same name are not visible to the current session while the temporary table exists, unless they are referenced with schema-qualified names. Any indexes created on a temporary table are automatically temporary as well.
+
+ *table\_name*
+The name (optionally schema-qualified) of the table to be created.
+
+ *column\_name*
+The name of a column to be created in the new table.
+
+ *data\_type*
+The data type of the column. This may include array specifiers.
+
+DEFAULT *default\_expr*
+The `DEFAULT` clause assigns a default data value for the column whose column definition it appears within. The value is any variable-free expression (subqueries and cross-references to other columns in the current table are not allowed). The data type of the default expression must match the data type of the column. The default expression will be used in any insert operation that does not specify a value for the column. If there is no default for a column, then the default is null.
+
+INHERITS
+The optional `INHERITS` clause specifies a list of tables from which the new table automatically inherits all columns. Use of `INHERITS` creates a persistent relationship between the new child table and its parent table(s). Schema modifications to the parent(s) normally propagate to children as well, and by default the data of the child table is included in scans of the parent(s).
+
+In HAWQ, the `INHERITS` clause is not used when creating partitioned tables. Although the concept of inheritance is used in partition hierarchies, the inheritance structure of a partitioned table is created using the PARTITION BY clause.
+
+If the same column name exists in more than one parent table, an error is reported unless the data types of the columns match in each of the parent tables. If there is no conflict, then the duplicate columns are merged to form a single column in the new table. If the column name list of the new table contains a column name that is also inherited, the data type must likewise match the inherited column(s), and the column definitions are merged into one. However, inherited and new column declarations of the same name need not specify identical constraints: all constraints provided from any declaration are merged together and all are applied to the new table. If the new table explicitly specifies a default value for the column, this default overrides any defaults from inherited declarations of the column. Otherwise, any parents that specify default values for the column must all specify the same default, or an error will be reported.
+
+LIKE *other\_table* \[{INCLUDING | EXCLUDING} {DEFAULTS | CONSTRAINTS}\]
+The `LIKE` clause specifies a table from which the new table automatically copies all column names, data types, not-null constraints, and distribution policy. Storage properties like append-only or partition structure are not copied. Unlike `INHERITS`, the new table and original table are completely decoupled after creation is complete.
+
+Default expressions for the copied column definitions will only be copied if `INCLUDING DEFAULTS` is specified. The default behavior is to exclude default expressions, resulting in the copied columns in the new table having null defaults.
+
+Not-null constraints are always copied to the new table. `CHECK` constraints will only be copied if `INCLUDING CONSTRAINTS` is specified; other types of constraints will *never* be copied. Also, no distinction is made between column constraints and table constraints — when constraints are requested, all check constraints are copied.
+
+Note also that unlike `INHERITS`, copied columns and constraints are not merged with similarly named columns and constraints. If the same name is specified explicitly or in another `LIKE` clause an error is signalled.
+
+NULL | NOT NULL
+Specifies if the column is or is not allowed to contain null values. `NULL` is the default.
+
+CHECK ( *expression* )
+The `CHECK` clause specifies an expression producing a Boolean result which new or updated rows must satisfy for an insert or update operation to succeed. Expressions evaluating to `TRUE` or `UNKNOWN` succeed. Should any row of an insert or update operation produce a `FALSE` result an error exception is raised and the insert or update does not alter the database. A check constraint specified as a column constraint should reference that column's value only, while an expression appearing in a table constraint may reference multiple columns. `CHECK` expressions cannot contain subqueries nor refer to variables other than columns of the current row.
+
+WITH ( *storage\_option=value* )
+The `WITH` clause can be used to set storage options for the table or its indexes. Note that you can also set storage parameters on a particular partition or subpartition by declaring the `WITH` clause in the partition specification.
+
+Note: You cannot create a table with both column encodings and compression parameters in a WITH clause.
+
+The following storage options are available:
+
+**APPENDONLY** — Set to `TRUE` to create the table as an append-only table. If `FALSE` is specified, an error message displays stating that heap tables are not supported.
+
+**BLOCKSIZE** — Set to the size, in bytes for each block in a table. The `BLOCKSIZE` must be between 8192 and 2097152 bytes, and be a multiple of 8192. The default is 32768.
+
+**bucketnum** — Set to the number of hash buckets to be used in creating a hash-distributed table, specified as an integer greater than 0 and no more than the value of `default_hash_table_bucket_number`. The default when the table is created is 6 times the segment count. However, explicitly setting the bucket number when creating a hash table is recommended.
+
+**ORIENTATION** — Set to `row` (the default) for row-oriented storage, or parquet. The parquet column-oriented format can be more efficient for large-scale queries. This option is only valid if `APPENDONLY=TRUE`. Heap-storage tables can only be row-oriented.
+
+**COMPRESSTYPE** — Set to `ZLIB` (the default) `SNAPPY`or `GZIP` to specify the type of compression used. SNAPPY and GZIP support parquet tables. ZLIB provides more compact compression ratios at lower speeds. This option is only valid if `APPENDONLY=TRUE`.
+
+**COMPRESSLEVEL** — For zlib compression of append-only tables, set to an integer value between 1 (fastest compression) to 9 (highest compression ratio). If not declared, the default is 1. This option is valid only if `APPENDONLY=TRUE`.
+
+**OIDS** — Set to `OIDS=FALSE` (the default) so that rows do not have object identifiers assigned to them. Do not enable OIDS when creating a table. On large tables, such as those in a typical HAWQ system, using OIDs for table rows can cause wrap-around of the 32-bit OID counter. Once the counter wraps around, OIDs can no longer be assumed to be unique, which not only makes them useless to user applications, but can also cause problems in the HAWQ system catalog tables. In addition, excluding OIDs from a table reduces the space required to store the table on disk by 4 bytes per row, slightly improving performance. OIDS are not allowed on partitioned tables.
+
+ON COMMIT
+The behavior of temporary tables at the end of a transaction block can be controlled using `ON COMMIT`. The three options are:
+
+**PRESERVE ROWS** - No special action is taken at the ends of transactions for temporary tables. This is the default behavior.
+
+**DELETE ROWS** - All rows in the temporary table will be deleted at the end of each transaction block. Essentially, an automatic `TRUNCATE` is done at each commit.
+
+**DROP** - The temporary table will be dropped at the end of the current transaction block.
+
+TABLESPACE *tablespace*
+The name of the tablespace in which the new table is to be created. If not specified, the database's default tablespace dfs\_default is used. Creating table on tablespace `pg_default` is not allowed.
+
+DISTRIBUTED BY (*column*, \[ ... \] )
+DISTRIBUTED RANDOMLY
+Used to declare the HAWQ distribution policy for the table. The default is RANDOM distribution. `DISTIBUTED BY` can use hash distribution with one or more columns declared as the distribution key. If hash distribution is desired, it must be specified using the first eligible column of the table as the distribution key.
+
+PARTITION BY
+Declares one or more columns by which to partition the table.
+
+ *partition\_type*
+Declares partition type: `LIST` (list of values) or `RANGE` (a numeric or date range).
+
+ *partition\_specification*
+Declares the individual partitions to create. Each partition can be defined individually or, for range partitions, you can use the `EVERY` clause (with a `START` and optional `END` clause) to define an increment pattern to use to create the individual partitions.
+
+**`DEFAULT PARTITION name `** — Declares a default partition. When data does not match to an existing partition, it is inserted into the default partition. Partition designs that do not have a default partition will reject incoming rows that do not match to an existing partition.
+
+**`PARTITION name `** — Declares a name to use for the partition. Partitions are created using the following naming convention: ` parentname_level#_prt_givenname `.
+
+**`VALUES`** — For list partitions, defines the value(s) that the partition will contain.
+
+**`START`** — For range partitions, defines the starting range value for the partition. By default, start values are `INCLUSIVE`. For example, if you declared a start date of '`2008-01-01`', then the partition would contain all dates greater than or equal to '`2008-01-01`'. Typically the data type of the `START` expression is the same type as the partition key column. If that is not the case, then you must explicitly cast to the intended data type.
+
+**`END`** — For range partitions, defines the ending range value for the partition. By default, end values are `EXCLUSIVE`. For example, if you declared an end date of '`2008-02-01`', then the partition would contain all dates less than but not equal to '`2008-02-01`'. Typically the data type of the `END` expression is the same type as the partition key column. If that is not the case, then you must explicitly cast to the intended data type.
+
+**`EVERY`** — For range partitions, defines how to increment the values from `START` to `END` to create individual partitions. Typically the data type of the `EVERY` expression is the same type as the partition key column. If that is not the case, then you must explicitly cast to the intended data type.
+
+**`WITH`** — Sets the table storage options for a partition. For example, you may want older partitions to be append-only tables and newer partitions to be regular heap tables.
+
+**`TABLESPACE`** — The name of the tablespace in which the partition is to be created.
+
+SUBPARTITION BY
+Declares one or more columns by which to subpartition the first-level partitions of the table. The format of the subpartition specification is similar to that of a partition specification described above.
+
+SUBPARTITION TEMPLATE
+Instead of declaring each subpartition definition individually for each partition, you can optionally declare a subpartition template to be used to create the subpartitions. This subpartition specification would then apply to all parent partitions.
+
+## Notes<a id="topic1__section5"></a>
+
+Using OIDs in new applications is not recommended. Avoid assuming that OIDs are unique across tables; if you need a database-wide unique identifier, use the combination of table OID and row OID for the purpose.
+
+Primary key and foreign key constraints are not supported in HAWQ. For inherited tables, table privileges *are not* inherited in the current implementation.
+
+HAWQ also supports the parquet columnar storage format. Parquet tables can be more efficient for increasing performance on large queries.
+
+## Setting Parameters for Parquet Tables<a id="parquetset"></a>
+
+You can set three kinds of parameters for a parquet table.
+
+1. Set the parquet orientation parameter:
+
+ ``` pre
+ with (appendonly=true, orientation=parquet);
+ ```
+
+2. Set the compression type parameter: Parquet tables can be compressed either as `SNAPPY` or `GZIP`. When setting to `SNAPPY`, setting a compression level causes it to fail. You can set the `GZIP` compression level between 1 and 9. If you specify a compression level but not a compression type when creating a parquet table, the compression type defaults to `GZIP`.
+
+ **Note:** The compress type `ZLIB` is valid for append-only tables, while `SNAPPY` and `GZIP` are valid for parquet tables.`QUICKLZ` compression has been deprecated.
+
+3. Set the data storage parameter: By default, the two parameters, `PAGESIZE` and `ROWGROUPSIZE` are set to 1MB/8MB for common and partitioned tables.
+
+ **Note:** The page size should be less than the rowgroup size. This is because rowgroup includes the metadata information of a single page even for a single column table. The parameters `PAGESIZE` and `ROWGROUPSIZE` are valid for parquet tables, while `BLOCKSIZE` is valid for append-only tables
+
+## About Parquet Storage<a id="aboutparquet"></a>
+
+DDL and DML: Most DDL and DML operations are valid for a parquet table. The usage for DDL and DML operations is similar to append-only tables. Valid operations on parquet tables include:
+
+- Parquet table creation (with/without partition, with/without compression type)
+- Insert and Select
+
+**Compression type and level**: You can only set the compression type at the table level. HAWQ does not support setting column level compression. The specified compression type is propagated to the columns. All the columns must have the same compress type and level.
+
+**Data type**: HAWQ supports all data types except arrays and user defined types.
+
+**Alter table**: HAWQ does not support adding a new column to an existing parquet table or dropping a column. You can use `ALTER TABLE` for a partition operation.
+
+**FillFactor/OIDS/Checksum**: HAWQ does not support these operations when creating parquet tables. The default value for checksum for a parquet table is false. You cannot set this value or specify fillfactor and oids.
+
+**Memory occupation**: When inserting or loading data to a parquet table, the whole rowgroup is stored in physical memory until the size exceeds the threshold or the end of the `INSERT` operation. Once either occurs, the entire rowgroup is flushed to disk. Also, at the beginning of the `INSERT` operation, each column is pre-allocated a page buffer. The column pre-allocated page buffer size should be `min(pageSizeLimit, rowgroupSizeLimit/estimatedColumnWidth/estimatedRecordWidth)` for the first rowgroup. For the following rowgroup, it should be `min(pageSizeLimit, actualColumnChunkSize in last rowgroup * 1.05)`, of which 1.05 is the estimated scaling factor. When reading data from a parquet table, the requested columns of the row group are loaded into memory. Memory is allocated 8 MB by default. Ensure that memory occupation does not exceed physical memory when setting `ROWGROUPSIZE` or `PAGESIZE`, otherwise you may encounter an out of memory error.
+
+## Parquet Examples<a id="parquetexamples"></a>
+
+**Parquet Example 1**
+
+Create an append-only table using the parquet format:
+
+``` pre
+CREATE TABLE customer ( id integer, fname text, lname text,
+ address text, city text, state text, zip text )
+WITH (APPENDONLY=true, ORIENTATION=parquet, OIDS=FALSE)
+DISTRIBUTED BY (id);
+```
+
+**Parquet Example 2**
+
+Create a parquet table with twelve monthly partitions:
+
+``` pre
+CREATE TABLE sales (id int, date date, amt decimal(10,2))
+WITH (APPENDONLY=true, ORIENTATION=parquet, OIDS=FALSE)
+DISTRIBUTED BY (id)
+PARTITION BY RANGE (date)
+ ( START (date '2016-01-01') INCLUSIVE
+ END (date '2017-01-01') EXCLUSIVE
+ EVERY (INTERVAL '1 month')
+ );
+```
+
+**Parquet Example 3**
+
+Add a new partition to the sales table:
+
+``` pre
+ALTER TABLE sales ADD PARTITION
+ START (date '2017-01-01') INCLUSIVE
+ END (date '2017-02-01') EXCLUSIVE;
+```
+
+## AO Examples<a id="aoexamples"></a>
+
+**AO Example 1**: Create a table named rank in the schema named baby and distribute the data using the columns rank, gender, and year:
+
+``` pre
+CREATE TABLE baby.rank ( id int, rank int, year smallint, gender char(1), count int )
+DISTRIBUTED BY (rank, gender, year);
+```
+
+**AO Example 2**: Create table films and table distributors. The first column will be used as the HAWQ distribution key by default:
+
+``` pre
+CREATE TABLE films (
+ code char(5), title varchar(40) NOT NULL, did integer NOT NULL,
+ date_prod date, kind varchar(10), len interval hour to minute
+);
+
+CREATE TABLE distributors (
+ did integer,
+ name varchar(40) NOT NULL CHECK (name <> '')
+);
+```
+
+**AO Example 3**: Create a gzip-compressed, append-only table:
+
+``` pre
+CREATE TABLE sales (txn_id int, qty int, date date)
+WITH (appendonly=true, compresslevel=5)
+DISTRIBUTED BY (txn_id);
+```
+
+**AO Example 4**: Create a three level partitioned table using subpartition templates and default partitions at each level:
+
+``` pre
+CREATE TABLE sales (id int, year int, month int, day int,
+region text)
+DISTRIBUTED BY (id)
+PARTITION BY RANGE (year)
+SUBPARTITION BY RANGE (month)
+SUBPARTITION TEMPLATE (
+START (1) END (13) EVERY (1),
+DEFAULT SUBPARTITION other_months )
+SUBPARTITION BY LIST (region)
+SUBPARTITION TEMPLATE (
+SUBPARTITION usa VALUES ('usa'),
+SUBPARTITION europe VALUES ('europe'),
+SUBPARTITION asia VALUES ('asia'),
+DEFAULT SUBPARTITION other_regions)
+( START (2002) END (2010) EVERY (1),
+DEFAULT PARTITION outlying_years);
+```
+
+**AO Example 5** Create a hash-distributed table named “sales” with 100 buckets.
+
+``` pre
+CREATE TABLE sales(id int, profit float)
+WITH (bucketnum=100)
+DISTRIBUTED BY (id);
+```
+
+## Compatibility<a id="topic1__section7"></a>
+
+The `CREATE TABLE` command conforms to the SQL standard, with the following exceptions:
+
+- **Temporary Tables** — In the SQL standard, temporary tables are defined just once and automatically exist (starting with empty contents) in every session that needs them. HAWQ instead requires each session to issue its own `CREATE TEMPORARY TABLE` command for each temporary table to be used. This allows different sessions to use the same temporary table name for different purposes, whereas the standard's approach constrains all instances of a given temporary table name to have the same table structure.
+
+ The standard's distinction between global and local temporary tables is not in HAWQ. HAWQ will accept the `GLOBAL` and `LOCAL` keywords in a temporary table declaration, but they have no effect.
+
+ If the `ON COMMIT` clause is omitted, the SQL standard specifies that the default behavior as `ON COMMIT DELETE ROWS`. However, the default behavior in HAWQ is `ON COMMIT PRESERVE ROWS`. The `ON COMMIT DROP` option does not exist in the SQL standard.
+
+- **Column Check Constraints** — The SQL standard says that `CHECK` column constraints may only refer to the column they apply to; only `CHECK` table constraints may refer to multiple columns. HAWQ does not enforce this restriction; it treats column and table check constraints alike.
+- **NULL Constraint** — The `NULL` constraint is a HAWQ extension to the SQL standard that is included for compatibility with some other database systems (and for symmetry with the `NOT NULL` constraint). Since it is the default for any column, its presence is not required.
+- **Inheritance** — Multiple inheritance via the `INHERITS` clause is a HAWQ language extension. SQL:1999 and later define single inheritance using a different syntax and different semantics. SQL:1999-style inheritance is not yet supported by HAWQ.
+- **Partitioning** — Table partitioning via the `PARTITION BY` clause is a HAWQ language extension.
+- **Zero-column tables** — HAWQ allows a table of no columns to be created (for example, `CREATE TABLE foo();`). This is an extension from the SQL standard, which does not allow zero-column tables. Zero-column tables are not in themselves very useful, but disallowing them creates odd special cases for `ALTER TABLE DROP COLUMN`, so this spec restriction is ignored.
+- **WITH clause** — The `WITH` clause is an extension; neither storage parameters nor OIDs are in the standard.
+- **Tablespaces** — The HAWQ concept of tablespaces is not part of the SQL standard. The clauses `TABLESPACE` and `USING INDEX TABLESPACE` are extensions.
+- **Data Distribution** — The HAWQ concept of a parallel or distributed database is not part of the SQL standard. The `DISTRIBUTED` clauses are extensions.
+
+## See Also<a id="topic1__section8"></a>
+
+[ALTER TABLE](ALTER-TABLE.html), [DROP TABLE](DROP-TABLE.html), [CREATE EXTERNAL TABLE](CREATE-EXTERNAL-TABLE.html), [CREATE TABLE AS](CREATE-TABLE-AS.html)
diff --git a/reference/sql/CREATE-TABLESPACE.html.md.erb b/reference/sql/CREATE-TABLESPACE.html.md.erb
new file mode 100644
index 0000000..52a7fb2
--- /dev/null
+++ b/reference/sql/CREATE-TABLESPACE.html.md.erb
@@ -0,0 +1,58 @@
+---
+title: CREATE TABLESPACE
+---
+
+Defines a new tablespace.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+CREATE TABLESPACE tablespace_name [OWNER username]
+ FILESPACE filespace_name
+
+```
+
+## Description<a id="topic1__section3"></a>
+
+`CREATE TABLESPACE` registers a new tablespace for your HAWQ system. The tablespace name must be distinct from the name of any existing tablespace in the system.
+
+A tablespace allows superusers to define an alternative location on the file system where the data files containing database objects (such as tables) may reside.
+
+A user with appropriate privileges can pass a tablespace name to [CREATE DATABASE](CREATE-DATABASE.html) or [CREATE TABLE](CREATE-TABLE.html) to have the data files for these objects stored within the specified tablespace.
+
+In HAWQ, there must be a file system location defined for the master and each segment in order for the tablespace to have a location to store its objects across an entire HAWQ system. This collection of file system locations is defined in a filespace object. A filespace must be defined before you can create a tablespace. See [hawq filespace](../cli/admin_utilities/hawqfilespace.html#topic1) for more information.
+
+## Parameters<a id="topic1__section4"></a>
+
+ *tablespacename*
+The name of a tablespace to be created. The name cannot begin with `pg_`, as such names are reserved for system tablespaces.
+
+OWNER *username*
+The name of the user who will own the tablespace. If omitted, defaults to the user executing the command. Only superusers may create tablespaces, but they can assign ownership of tablespaces to non-superusers.
+
+FILESPACE *filespace\_name*
+The name of a HAWQ filespace that was defined using the `hawq filespace` management utility.
+
+## Notes<a id="topic1__section5"></a>
+
+You must first create a filespace to be used by the tablespace. See [hawq filespace](../cli/admin_utilities/hawqfilespace.html#topic1) for more information.
+
+Tablespaces are only supported on systems that support symbolic links.
+
+`CREATE TABLESPACE` cannot be executed inside a transaction block.
+
+## Examples<a id="topic1__section6"></a>
+
+Create a new tablespace by specifying the corresponding filespace to use:
+
+``` pre
+CREATE TABLESPACE mytblspace FILESPACE myfilespace;
+```
+
+## Compatibility<a id="topic1__section7"></a>
+
+`CREATE TABLESPACE` is a HAWQ extension.
+
+## See Also<a id="topic1__section8"></a>
+
+[CREATE DATABASE](CREATE-DATABASE.html), [CREATE TABLE](CREATE-TABLE.html), [DROP TABLESPACE](DROP-TABLESPACE.html)
diff --git a/reference/sql/CREATE-TYPE.html.md.erb b/reference/sql/CREATE-TYPE.html.md.erb
new file mode 100644
index 0000000..5e49553
--- /dev/null
+++ b/reference/sql/CREATE-TYPE.html.md.erb
@@ -0,0 +1,187 @@
+---
+title: CREATE TYPE
+---
+
+Defines a new data type.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+CREATE TYPE name AS ( attribute_name
+ data_type [, ... ] )
+
+CREATE TYPE name (
+ INPUT = input_function,
+ OUTPUT = output_function
+ [, RECEIVE = receive_function]
+ [, SEND = send_function]
+ [, INTERNALLENGTH = {internallength | VARIABLE}]
+ [, PASSEDBYVALUE]
+ [, ALIGNMENT = alignment]
+ [, STORAGE = storage]
+ [, DEFAULT = default]
+ [, ELEMENT = element]
+ [, DELIMITER = delimiter] )
+
+CREATE TYPE name
+
+```
+
+## Description<a id="topic1__section3"></a>
+
+`CREATE TYPE` registers a new data type for use in the current database. The user who defines a type becomes its owner.
+
+If a schema name is given then the type is created in the specified schema. Otherwise it is created in the current schema. The type name must be distinct from the name of any existing type or domain in the same schema. The type name must also be distinct from the name of any existing table in the same schema.
+
+**Composite Types**
+
+The first form of `CREATE TYPE` creates a composite type. This is the only form currently supported by HAWQ. The composite type is specified by a list of attribute names and data types. This is essentially the same as the row type of a table, but using `CREATE TYPE` avoids the need to create an actual table when all that is wanted is to define a type. A stand-alone composite type is useful as the argument or return type of a function.
+
+**Base Types**
+
+The second form of `CREATE TYPE` creates a new base type (scalar type). The parameters may appear in any order, not only that shown in the syntax, and most are optional. You must register two or more functions (using `CREATE FUNCTION`) before defining the type. The support functions *input\_function* and *output\_function* are required, while the functions *receive\_function*, *send\_function* and *analyze\_function* are optional. Generally these functions have to be coded in C or another low-level language. In HAWQ, any function used to implement a data type must be defined as `IMMUTABLE`.
+
+The *input\_function* converts the type's external textual representation to the internal representation used by the operators and functions defined for the type. *output\_function* performs the reverse transformation. The input function may be declared as taking one argument of type `cstring`, or as taking three arguments of types `cstring`, `oid`, `integer`. The first argument is the input text as a C string, the second argument is the type's own OID (except for array types, which instead receive their element type's OID), and the third is the `typmod` of the destination column, if known (`-1` will be passed if not). The input function must return a value of the data type itself. Usually, an input function should be declared `STRICT`; if it is not, it will be called with a `NULL` first parameter when reading a `NULL` input value. The function must still return `NULL` in this case, unless it raises an error. (This case is mainly meant to support domain input functions, which may need to reject `NULL` inputs.) The output function must be declared as taking one argument of the new data type. The output function must return type `cstring`. Output functions are not invoked for `NULL` values.
+
+The optional *receive\_function* converts the type's external binary representation to the internal representation. If this function is not supplied, the type cannot participate in binary input. The binary representation should be chosen to be cheap to convert to internal form, while being reasonably portable. (For example, the standard integer data types use network byte order as the external binary representation, while the internal representation is in the machine's native byte order.) The receive function should perform adequate checking to ensure that the value is valid. The receive function may be declared as taking one argument of type `internal`, or as taking three arguments of types `internal`, `oid`, `integer`. The first argument is a pointer to a `StringInfo` buffer holding the received byte string; the optional arguments are the same as for the text input function. The receive function must return a value of the data type itself. Usually, a receive function should be declared `STRICT`; if it is not, it will be called with a `NULL` first parameter when reading a NULL input value. The function must still return `NULL` in this case, unless it raises an error. (This case is mainly meant to support domain receive functions, which may need to reject `NULL` inputs.) Similarly, the optional *send\_function* converts from the internal representation to the external binary representation. If this function is not supplied, the type cannot participate in binary output. The send function must be declared as taking one argument of the new data type. The send function must return type `bytea`. Send functions are not invoked for `NULL` values.
+
+You should at this point be wondering how the input and output functions can be declared to have results or arguments of the new type, when they have to be created before the new type can be created. The answer is that the type should first be defined as a shell type, which is a placeholder type that has no properties except a name and an owner. This is done by issuing the command `CREATE TYPE name `, with no additional parameters. Then the I/O functions can be defined referencing the shell type. Finally, `CREATE TYPE` with a full definition replaces the shell entry with a complete, valid type definition, after which the new type can be used normally.
+
+While the details of the new type's internal representation are only known to the I/O functions and other functions you create to work with the type, there are several properties of the internal representation that must be declared to HAWQ. Foremost of these is *internallength*. Base data types can be fixed-length, in which case *internallength* is a positive integer, or variable length, indicated by setting *internallength* to `VARIABLE`. (Internally, this is represented by setting `typlen` to `-1`.) The internal representation of all variable-length types must start with a 4-byte integer giving the total length of this value of the type.
+
+The optional flag `PASSEDBYVALUE` indicates that values of this data type are passed by value, rather than by reference. You may not pass by value types whose internal representation is larger than the size of the `Datum` type (4 bytes on most machines, 8 bytes on a few).
+
+The *alignment* parameter specifies the storage alignment required for the data type. The allowed values equate to alignment on 1, 2, 4, or 8 byte boundaries. Note that variable-length types must have an alignment of at least 4, since they necessarily contain an `int4` as their first component.
+
+The *storage* parameter allows selection of storage strategies for variable-length data types. (Only `plain` is allowed for fixed-length types.) `plain` specifies that data of the type will always be stored in-line and not compressed. `extended` specifies that the system will first try to compress a long data value, and will move the value out of the main table row if it's still too long. `external` allows the value to be moved out of the main table, but the system will not try to compress it. `main` allows compression, but discourages moving the value out of the main table. (Data items with this storage strategy may still be moved out of the main table if there is no other way to make a row fit, but they will be kept in the main table preferentially over `extended` and `external` items.)
+
+A default value may be specified, in case a user wants columns of the data type to default to something other than the null value. Specify the default with the `DEFAULT` key word. (Such a default may be overridden by an explicit `DEFAULT` clause attached to a particular column.)
+
+To indicate that a type is an array, specify the type of the array elements using the `ELEMENT` key word. For example, to define an array of 4-byte integers (int4), specify `ELEMENT = int4`. More details about array types appear below.
+
+To indicate the delimiter to be used between values in the external representation of arrays of this type, `delimiter` can be set to a specific character. The default delimiter is the comma (,). Note that the delimiter is associated with the array element type, not the array type itself.
+
+**Array Types**
+
+Whenever a user-defined base data type is created, HAWQ automatically creates an associated array type, whose name consists of the base type's name prepended with an underscore. The parser understands this naming convention, and translates requests for columns of type `foo[]` into requests for type `_foo`. The implicitly-created array type is variable length and uses the built-in input and output functions `array_in` and `array_out`.
+
+You might reasonably ask why there is an `ELEMENT` option, if the system makes the correct array type automatically. The only case where it's useful to use `ELEMENT` is when you are making a fixed-length type that happens to be internally an array of a number of identical things, and you want to allow these things to be accessed directly by subscripting, in addition to whatever operations you plan to provide for the type as a whole. For example, type `name` allows its constituent `char` elements to be accessed this way. A 2-D point type could allow its two component numbers to be accessed like point\[0\] and point\[1\]. Note that this facility only works for fixed-length types whose internal form is exactly a sequence of identical fixed-length fields. A subscriptable variable-length type must have the generalized internal representation used by `array_in` and `array_out`. For historical reasons, subscripting of fixed-length array types starts from zero, rather than from one as for variable-length arrays.
+
+## Parameters<a id="topic1__section7"></a>
+
+ *name*
+The name (optionally schema-qualified) of a type to be created.
+
+ *attribute\_name*
+The name of an attribute (column) for the composite type.
+
+ *data\_type*
+The name of an existing data type to become a column of the composite type.
+
+ *input\_function*
+The name of a function that converts data from the type's external textual form to its internal form.
+
+ *output\_function*
+The name of a function that converts data from the type's internal form to its external textual form.
+
+ *receive\_function*
+The name of a function that converts data from the type's external binary form to its internal form.
+
+ *send\_function*
+The name of a function that converts data from the type's internal form to its external binary form.
+
+ *internallength*
+A numeric constant that specifies the length in bytes of the new type's internal representation. The default assumption is that it is variable-length.
+
+ *alignment*
+The storage alignment requirement of the data type. Must be one of `char`, `int2`, `int4`, or `double`. The default is `int4`.
+
+ *storage*
+The storage strategy for the data type. Must be one of `plain`, `external`, `extended`, or `main`. The default is `plain`.
+
+ *default*
+The default value for the data type. If this is omitted, the default is null.
+
+ *element*
+The type being created is an array; this specifies the type of the array elements.
+
+ *delimiter*
+The delimiter character to be used between values in arrays made of this type.
+
+## Notes<a id="topic1__section8"></a>
+
+User-defined type names cannot begin with the underscore character (\_) and can only be 62 characters long (or in general `NAMEDATALEN - 2`, rather than the `NAMEDATALEN - 1` characters allowed for other names). Type names beginning with underscore are reserved for internally-created array type names.
+
+Because there are no restrictions on use of a data type once it's been created, creating a base type is tantamount to granting public execute permission on the functions mentioned in the type definition. (The creator of the type is therefore required to own these functions.) This is usually not an issue for the sorts of functions that are useful in a type definition. But you might want to think twice before designing a type in a way that would require 'secret' information to be used while converting it to or from external form.
+
+Before HAWQ version 2.4, the syntax `CREATE TYPE name ` did not exist. The way to create a new base type was to create its input function first. In this approach, HAWQ will first see the name of the new data type as the return type of the input function. The shell type is implicitly created in this situation, and then it can be referenced in the definitions of the remaining I/O functions. This approach still works, but is deprecated and may be disallowed in some future release. Also, to avoid accidentally cluttering the catalogs with shell types as a result of simple typos in function definitions, a shell type will only be made this way when the input function is written in C.
+
+## Examples<a id="topic1__section9"></a>
+
+This example creates a composite type and uses it in a function definition:
+
+``` pre
+CREATE TYPE compfoo AS (f1 int, f2 text);
+
+CREATE FUNCTION getfoo() RETURNS SETOF compfoo AS $$
+ SELECT fooid, fooname FROM foo
+$$ LANGUAGE SQL;
+```
+
+This example creates the base data type `box` and then uses the type in a table definition:
+
+``` pre
+CREATE TYPE box;
+
+CREATE FUNCTION my_box_in_function(cstring) RETURNS box AS
+... ;
+
+CREATE FUNCTION my_box_out_function(box) RETURNS cstring AS
+... ;
+
+CREATE TYPE box (
+ INTERNALLENGTH = 16,
+ INPUT = my_box_in_function,
+ OUTPUT = my_box_out_function
+);
+
+CREATE TABLE myboxes (
+ id integer,
+ description box
+);
+```
+
+If the internal structure of `box` were an array of four `float4` elements, we might instead use:
+
+``` pre
+CREATE TYPE box (
+ INTERNALLENGTH = 16,
+ INPUT = my_box_in_function,
+ OUTPUT = my_box_out_function,
+ ELEMENT = float4
+);
+```
+
+which would allow a box value's component numbers to be accessed by subscripting. Otherwise the type behaves the same as before.
+
+This example creates a large object type and uses it in a table definition:
+
+``` pre
+CREATE TYPE bigobj (
+ INPUT = lo_filein, OUTPUT = lo_fileout,
+ INTERNALLENGTH = VARIABLE
+);
+
+CREATE TABLE big_objs (
+ id integer,
+ obj bigobj
+);
+```
+
+## Compatibility<a id="topic1__section10"></a>
+
+`CREATE TYPE` command is a HAWQ extension. There is a `CREATE TYPE` statement in the SQL standard that is rather different in detail.
+
+## See Also<a id="topic1__section11"></a>
+
+[CREATE FUNCTION](CREATE-FUNCTION.html), [ALTER TYPE](ALTER-TYPE.html), [DROP TYPE](DROP-TYPE.html)
diff --git a/reference/sql/CREATE-USER.html.md.erb b/reference/sql/CREATE-USER.html.md.erb
new file mode 100644
index 0000000..d4a76da
--- /dev/null
+++ b/reference/sql/CREATE-USER.html.md.erb
@@ -0,0 +1,46 @@
+---
+title: CREATE USER
+---
+
+Defines a new database role with the `LOGIN` privilege by default.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+CREATE USER name [ [WITH] option [ ... ] ]
+```
+
+where *option* can be:
+
+``` pre
+ SUPERUSER | NOSUPERUSER
+ | CREATEDB | NOCREATEDB
+ | CREATEROLE | NOCREATEROLE
+ | CREATEUSER | NOCREATEUSER
+ | INHERIT | NOINHERIT
+ | LOGIN | NOLOGIN
+ | [ ENCRYPTED | UNENCRYPTED ] PASSWORD 'password'
+ | VALID UNTIL 'timestamp'
+ | IN ROLE rolename [, ...]
+ | IN GROUP rolename [, ...]
+ | ROLE rolename [, ...]
+ | ADMIN rolename [, ...]
+ | USER rolename [, ...]
+ | SYSID uid
+ | RESOURCE QUEUE queue_name
+
+```
+
+## Description<a id="topic1__section3"></a>
+
+HAWQ does not support `CREATE USER`. This command has been replaced by [CREATE ROLE](CREATE-ROLE.html).
+
+The only difference between `CREATE ROLE` and `CREATE USER` is that `LOGIN` is assumed by default with `CREATE USER`, whereas `NOLOGIN` is assumed by default with `CREATE ROLE`.
+
+## Compatibility<a id="topic1__section4"></a>
+
+There is no `CREATE USER` statement in the SQL standard.
+
+## See Also<a id="topic1__section5"></a>
+
+[CREATE ROLE](CREATE-ROLE.html)
diff --git a/reference/sql/CREATE-VIEW.html.md.erb b/reference/sql/CREATE-VIEW.html.md.erb
new file mode 100644
index 0000000..2595843
--- /dev/null
+++ b/reference/sql/CREATE-VIEW.html.md.erb
@@ -0,0 +1,88 @@
+---
+title: CREATE VIEW
+---
+
+Defines a new view.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+CREATE [OR REPLACE] [TEMP | TEMPORARY] VIEW name
+ [ ( column_name [, ...] ) ]
+ AS query
+
+```
+
+## Description<a id="topic1__section3"></a>
+
+`CREATE VIEW` defines a view of a query. The view is not physically materialized. Instead, the query is run every time the view is referenced in a query.
+
+`CREATE OR REPLACE VIEW` is similar, but if a view of the same name already exists, it is replaced. You can only replace a view with a new query that generates the identical set of columns (same column names and data types).
+
+If a schema name is given then the view is created in the specified schema. Otherwise it is created in the current schema. Temporary views exist in a special schema, so a schema name may not be given when creating a temporary view. The name of the view must be distinct from the name of any other view, table, sequence, or index in the same schema.
+
+## Parameters<a id="topic1__section4"></a>
+
+TEMPORARY | TEMP
+If specified, the view is created as a temporary view. Temporary views are automatically dropped at the end of the current session. Existing permanent relations with the same name are not visible to the current session while the temporary view exists, unless they are referenced with schema-qualified names. If any of the tables referenced by the view are temporary, the view is created as a temporary view (whether `TEMPORARY` is specified or not).
+
+ *name*
+The name (optionally schema-qualified) of a view to be created.
+
+ *column\_name*
+An optional list of names to be used for columns of the view. If not given, the column names are deduced from the query.
+
+ *query*
+A [SELECT](SELECT.html) command which will provide the columns and rows of the view.
+
+## Notes<a id="topic1__section5"></a>
+
+Views in HAWQ are read only. The system will not allow an insert, update, or delete on a view. You can get the effect of an updatable view by creating rewrite rules on the view into appropriate actions on other tables. For more information see `CREATE RULE`.
+
+Be careful that the names and data types of the view's columns will be assigned the way you want. For example, if you run the following command:
+
+``` pre
+CREATE VIEW vista AS SELECT 'Hello World';
+```
+
+The result is poor: the column name defaults to `?column?`, and the column data type defaults to `unknown`. If you want a string literal in a view's result, use the following command:
+
+``` pre
+CREATE VIEW vista AS SELECT text 'Hello World' AS hello;
+```
+
+Check that you have permission to access the tables referenced in the view. View ownership determines permissions, not your status as current user. This is true, even if you are a superuser. This concept is unusual, since superusers typically have access to all objects. In the case of views, even superusers must be explicitly granted access to tables referenced if they do not own the view.
+
+However, functions called in the view are treated the same as if they had been called directly from the query using the view. Therefore the user of a view must have permissions to call any functions used by the view.
+
+If you create a view with an `ORDER BY` clause, the `ORDER BY` clause is ignored when you do a `SELECT` from the view.
+
+## Examples<a id="topic1__section6"></a>
+
+Create a view consisting of all comedy films:
+
+``` pre
+CREATE VIEW comedies AS SELECT * FROM films WHERE kind =
+'comedy';
+```
+
+Create a view that gets the top ten ranked baby names:
+
+``` pre
+CREATE VIEW topten AS SELECT name, rank, gender, year FROM
+names, rank WHERE rank < '11' AND names.id=rank.id;
+```
+
+## Compatibility<a id="topic1__section7"></a>
+
+The SQL standard specifies some additional capabilities for the `CREATE VIEW` statement that are not in HAWQ. The optional clauses for the full SQL command in the standard are:
+
+- **CHECK OPTION** — This option has to do with updatable views. All `INSERT` and `UPDATE` commands on the view will be checked to ensure data satisfy the view-defining condition (that is, the new data would be visible through the view). If they do not, the update will be rejected.
+- **LOCAL** — Check for integrity on this view.
+- **CASCADED** — Check for integrity on this view and on any dependent view. `CASCADED` is assumed if neither `CASCADED` nor `LOCAL` is specified.
+
+`CREATE OR REPLACE VIEW` is a HAWQ language extension. So is the concept of a temporary view.
+
+## See Also<a id="topic1__section8"></a>
+
+[SELECT](SELECT.html), [DROP VIEW](DROP-VIEW.html)
diff --git a/reference/sql/DEALLOCATE.html.md.erb b/reference/sql/DEALLOCATE.html.md.erb
new file mode 100644
index 0000000..66f93fc
--- /dev/null
+++ b/reference/sql/DEALLOCATE.html.md.erb
@@ -0,0 +1,42 @@
+---
+title: DEALLOCATE
+---
+
+Deallocates a prepared statement.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+DEALLOCATE [PREPARE] name
+
+```
+
+## Description<a id="topic1__section3"></a>
+
+`DEALLOCATE` is used to deallocate a previously prepared SQL statement. If you do not explicitly deallocate a prepared statement, it is deallocated when the session ends.
+
+For more information on prepared statements, see [PREPARE](PREPARE.html).
+
+## Parameters<a id="topic1__section4"></a>
+
+PREPARE
+Optional key word which is ignored.
+
+ *name*
+The name of the prepared statement to deallocate.
+
+## Examples<a id="topic1__section5"></a>
+
+Deallocated the previously prepared statement named `insert_names`:
+
+``` pre
+DEALLOCATE insert_names;
+```
+
+## Compatibility<a id="topic1__section6"></a>
+
+The SQL standard includes a `DEALLOCATE` statement, but it is only for use in embedded SQL.
+
+## See Also<a id="topic1__section7"></a>
+
+[EXECUTE](EXECUTE.html), [PREPARE](PREPARE.html)
diff --git a/reference/sql/DECLARE.html.md.erb b/reference/sql/DECLARE.html.md.erb
new file mode 100644
index 0000000..3c65c75
--- /dev/null
+++ b/reference/sql/DECLARE.html.md.erb
@@ -0,0 +1,84 @@
+---
+title: DECLARE
+---
+
+Defines a cursor.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+DECLARE name [BINARY] [INSENSITIVE] [NO SCROLL] CURSOR
+ [{WITH | WITHOUT} HOLD]
+ FOR query [FOR READ ONLY]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`DECLARE` allows a user to create cursors, which can be used to retrieve a small number of rows at a time out of a larger query. Cursors can return data either in text or in binary format using [FETCH](FETCH.html).
+
+Normal cursors return data in text format, the same as a `SELECT` would produce. Since data is stored natively in binary format, the system must do a conversion to produce the text format. Once the information comes back in text form, the client application may need to convert it to a binary format to manipulate it. In addition, data in the text format is often larger in size than in the binary format. Binary cursors return the data in a binary representation that may be more easily manipulated. Nevertheless, if you intend to display the data as text anyway, retrieving it in text form will save you some effort on the client side.
+
+As an example, if a query returns a value of one from an integer column, you would get a string of 1 with a default cursor whereas with a binary cursor you would get a 4-byte field containing the internal representation of the value (in big-endian byte order).
+
+Binary cursors should be used carefully. Many applications, including psql, are not prepared to handle binary cursors and expect data to come back in the text format.
+
+**Note:**
+When the client application uses the 'extended query' protocol to issue a `FETCH` command, the Bind protocol message specifies whether data is to be retrieved in text or binary format. This choice overrides the way that the cursor is defined. The concept of a binary cursor as such is thus obsolete when using extended query protocol — any cursor can be treated as either text or binary.
+
+## Parameters<a id="topic1__section4"></a>
+
+ *name*
+The name of the cursor to be created.
+
+BINARY
+Causes the cursor to return data in binary rather than in text format.
+
+INSENSITIVE
+Indicates that data retrieved from the cursor should be unaffected by updates to the tables underlying the cursor while the cursor exists. In HAWQ, all cursors are insensitive. This key word currently has no effect and is present for compatibility with the SQL standard.
+
+NO SCROLL
+A cursor cannot be used to retrieve rows in a nonsequential fashion. This is the default behavior in HAWQ, since scrollable cursors (`SCROLL`) are not supported.
+
+WITH HOLD
+WITHOUT HOLD
+`WITH HOLD` specifies that the cursor may continue to be used after the transaction that created it successfully commits. `WITHOUT HOLD` specifies that the cursor cannot be used outside of the transaction that created it. `WITHOUT HOLD` is the default.
+
+ *query*
+A [SELECT](SELECT.html) command which will provide the rows to be returned by the cursor.
+
+<!-- -->
+
+FOR READ ONLY
+`FOR READ ONLY` indicates that the cursor is used in a read-only mode. Cursors can only be used in a read-only mode in HAWQ. HAWQ does not support updatable cursors (FOR UPDATE), so this is the default behavior.
+
+## Notes<a id="topic1__section5"></a>
+
+Unless `WITH HOLD` is specified, the cursor created by this command can only be used within the current transaction. Thus, `DECLARE` without `WITH HOLD` is useless outside a transaction block: the cursor would survive only to the completion of the statement. Therefore HAWQ reports an error if this command is used outside a transaction block. Use `BEGIN`, `COMMIT` and `ROLLBACK` to define a transaction block.
+
+If `WITH HOLD` is specified and the transaction that created the cursor successfully commits, the cursor can continue to be accessed by subsequent transactions in the same session. (But if the creating transaction is aborted, the cursor is removed.) A cursor created with `WITH HOLD` is closed when an explicit `CLOSE` command is issued on it, or the session ends. In the current implementation, the rows represented by a held cursor are copied into a temporary file or memory area so that they remain available for subsequent transactions.
+
+Scrollable cursors are not currently supported in HAWQ. You can only use `FETCH` to move the cursor position forward, not backwards.
+
+You can see all available cursors by querying the `pg_cursors` system view.
+
+## Examples<a id="topic1__section6"></a>
+
+Declare a cursor:
+
+``` pre
+DECLARE mycursor CURSOR FOR SELECT * FROM mytable;
+```
+
+## Compatibility<a id="topic1__section7"></a>
+
+SQL standard allows cursors only in embedded SQL and in modules. HAWQ permits cursors to be used interactively.
+
+HAWQ does not implement an `OPEN` statement for cursors. A cursor is considered to be open when it is declared.
+
+The SQL standard allows cursors to move both forward and backward. All HAWQ cursors are forward moving only (not scrollable).
+
+Binary cursors are a HAWQ extension.
+
+## See Also<a id="topic1__section8"></a>
+
+[CLOSE](CLOSE.html), [FETCH](FETCH.html), [SELECT](SELECT.html)
diff --git a/reference/sql/DROP-AGGREGATE.html.md.erb b/reference/sql/DROP-AGGREGATE.html.md.erb
new file mode 100644
index 0000000..d373953
--- /dev/null
+++ b/reference/sql/DROP-AGGREGATE.html.md.erb
@@ -0,0 +1,48 @@
+---
+title: DROP AGGREGATE
+---
+
+Removes an aggregate function.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+DROP AGGREGATE [IF EXISTS] name ( type [, ...] ) [CASCADE | RESTRICT]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`DROP AGGREGATE` will delete an existing aggregate function. To execute this command the current user must be the owner of the aggregate function.
+
+## Parameters<a id="topic1__section4"></a>
+
+IF EXISTS
+Do not throw an error if the aggregate does not exist. A notice is issued in this case.
+
+ *name*
+The name (optionally schema-qualified) of an existing aggregate function.
+
+ *type*
+An input data type on which the aggregate function operates. To reference a zero-argument aggregate function, write `*` in place of the list of input data types.
+
+CASCADE
+Automatically drop objects that depend on the aggregate function.
+
+RESTRICT
+Refuse to drop the aggregate function if any objects depend on it. This is the default.
+
+## Examples<a id="topic1__section5"></a>
+
+To remove the aggregate function `myavg` for type `integer`:
+
+``` pre
+DROP AGGREGATE myavg(integer);
+```
+
+## Compatibility<a id="topic1__section6"></a>
+
+There is no `DROP AGGREGATE` statement in the SQL standard.
+
+## See Also<a id="topic1__section7"></a>
+
+[ALTER AGGREGATE](ALTER-AGGREGATE.html), [CREATE AGGREGATE](CREATE-AGGREGATE.html)
diff --git a/reference/sql/DROP-DATABASE.html.md.erb b/reference/sql/DROP-DATABASE.html.md.erb
new file mode 100644
index 0000000..e8d14ef
--- /dev/null
+++ b/reference/sql/DROP-DATABASE.html.md.erb
@@ -0,0 +1,48 @@
+---
+title: DROP DATABASE
+---
+
+Removes a database.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+DROP DATABASE [IF EXISTS] name
+
+```
+
+## Description<a id="topic1__section3"></a>
+
+`DROP DATABASE` drops a database. It removes the catalog entries for the database and deletes the directory containing the data. It can only be executed by the database owner. Also, it cannot be executed while you or anyone else are connected to the target database. (Connect to `template1` or any other database to issue this command.)
+
+**Warning:** `DROP DATABASE` cannot be undone. Use it with care!
+
+## Parameters<a id="topic1__section4"></a>
+
+IF EXISTS
+Do not throw an error if the database does not exist. A notice is issued in this case.
+
+ *name*
+The name of the database to remove.
+
+## Notes<a id="topic1__section5"></a>
+
+`DROP DATABASE` cannot be executed inside a transaction block.
+
+This command cannot be executed while connected to the target database. Thus, it might be more convenient to use the program `dropdb` instead, which is a wrapper around this command.
+
+## Examples<a id="topic1__section6"></a>
+
+Drop the database named `testdb`:
+
+``` pre
+DROP DATABASE testdb;
+```
+
+## Compatibility<a id="topic1__section7"></a>
+
+There is no `DROP DATABASE` statement in the SQL standard.
+
+## See Also<a id="topic1__section8"></a>
+
+[CREATE DATABASE](CREATE-DATABASE.html)
diff --git a/reference/sql/DROP-EXTERNAL-TABLE.html.md.erb b/reference/sql/DROP-EXTERNAL-TABLE.html.md.erb
new file mode 100644
index 0000000..e5e2fb8
--- /dev/null
+++ b/reference/sql/DROP-EXTERNAL-TABLE.html.md.erb
@@ -0,0 +1,48 @@
+---
+title: DROP EXTERNAL TABLE
+---
+
+Removes an external table definition.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+DROP EXTERNAL [WEB] TABLE [IF EXISTS] name [CASCADE | RESTRICT]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`DROP EXTERNAL TABLE` drops an existing external table definition from the database system. The external data sources or files are not deleted. To execute this command you must be the owner of the external table.
+
+## Parameters<a id="topic1__section4"></a>
+
+WEB
+Optional keyword for dropping external web tables.
+
+IF EXISTS
+Do not throw an error if the external table does not exist. A notice is issued in this case.
+
+ *name*
+The name (optionally schema-qualified) of an existing external table.
+
+CASCADE
+Automatically drop objects that depend on the external table (such as views).
+
+RESTRICT
+Refuse to drop the external table if any objects depend on it. This is the default.
+
+## Examples<a id="topic1__section5"></a>
+
+Remove the external table named `staging` if it exists:
+
+``` pre
+DROP EXTERNAL TABLE IF EXISTS staging;
+```
+
+## Compatibility<a id="topic1__section6"></a>
+
+There is no `DROP EXTERNAL TABLE` statement in the SQL standard.
+
+## See Also<a id="topic1__section7"></a>
+
+[CREATE EXTERNAL TABLE](CREATE-EXTERNAL-TABLE.html)
diff --git a/reference/sql/DROP-FILESPACE.html.md.erb b/reference/sql/DROP-FILESPACE.html.md.erb
new file mode 100644
index 0000000..23f8b53
--- /dev/null
+++ b/reference/sql/DROP-FILESPACE.html.md.erb
@@ -0,0 +1,42 @@
+---
+title: DROP FILESPACE
+---
+
+Removes a filespace.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+DROP FILESPACE [IF EXISTS] filespacename
+
+```
+
+## Description<a id="topic1__section3"></a>
+
+`DROP FILESPACE` removes a filespace definition and its system-generated data directories from the system.
+
+A filespace can only be dropped by its owner or a superuser. The filespace must be empty of all tablespace objects before it can be dropped. It is possible that tablespaces in other databases may still be using a filespace even if no tablespaces in the current database are using the filespace.
+
+## Parameters<a id="topic1__section4"></a>
+
+IF EXISTS
+Do not throw an error if the filespace does not exist. A notice is issued in this case.
+
+ *tablespacename*
+The name of the filespace to remove.
+
+## Examples<a id="topic1__section5"></a>
+
+Remove the tablespace `myfs`:
+
+``` pre
+DROP FILESPACE myfs;
+```
+
+## Compatibility<a id="topic1__section6"></a>
+
+There is no `DROP FILESPACE` statement in the SQL standard or in PostgreSQL.
+
+## See Also<a id="topic1__section7"></a>
+
+[DROP TABLESPACE](DROP-TABLESPACE.html), [hawq filespace](../cli/admin_utilities/hawqfilespace.html#topic1)
diff --git a/reference/sql/DROP-FUNCTION.html.md.erb b/reference/sql/DROP-FUNCTION.html.md.erb
new file mode 100644
index 0000000..383749e
--- /dev/null
+++ b/reference/sql/DROP-FUNCTION.html.md.erb
@@ -0,0 +1,55 @@
+---
+title: DROP FUNCTION
+---
+
+Removes a function.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+DROP FUNCTION [IF EXISTS] name ( [ [argmode] [argname] argtype
+ [, ...] ] ) [CASCADE | RESTRICT]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`DROP FUNCTION` removes the definition of an existing function. To execute this command the user must be the owner of the function. The argument types to the function must be specified, since several different functions may exist with the same name and different argument lists.
+
+## Parameters<a id="topic1__section4"></a>
+
+IF EXISTS
+Do not throw an error if the function does not exist. A notice is issued in this case.
+
+ *name*
+The name (optionally schema-qualified) of an existing function.
+
+ *argmode*
+The mode of an argument: either `IN`, `OUT`, or `INOUT`. If omitted, the default is IN. Note that `DROP FUNCTION` does not actually pay any attention to `OUT` arguments, since only the input arguments are needed to determine the function's identity. So it is sufficient to list the `IN` and `INOUT` arguments.
+
+ *argname*
+The name of an argument. Note that `DROP FUNCTION` does not actually pay any attention to argument names, since only the argument data types are needed to determine the function's identity.
+
+ *argtype*
+The data type(s) of the function's arguments (optionally schema-qualified), if any.
+
+CASCADE
+Automatically drop objects that depend on the function such as operators.
+
+RESTRICT
+Refuse to drop the function if any objects depend on it. This is the default.
+
+## Examples<a id="topic1__section5"></a>
+
+Drop the square root function:
+
+``` pre
+DROP FUNCTION sqrt(integer);
+```
+
+## Compatibility<a id="topic1__section6"></a>
+
+A `DROP FUNCTION` statement is defined in the SQL standard, but it is not compatible with this command.
+
+## See Also<a id="topic1__section7"></a>
+
+[ALTER FUNCTION](ALTER-FUNCTION.html), [CREATE FUNCTION](CREATE-FUNCTION.html)
diff --git a/reference/sql/DROP-GROUP.html.md.erb b/reference/sql/DROP-GROUP.html.md.erb
new file mode 100644
index 0000000..3a32505
--- /dev/null
+++ b/reference/sql/DROP-GROUP.html.md.erb
@@ -0,0 +1,31 @@
+---
+title: DROP GROUP
+---
+
+Removes a database role.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+DROP GROUP [IF EXISTS] name [, ...]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`DROP GROUP` is an obsolete command, though still accepted for backwards compatibility. Groups (and users) have been superseded by the more general concept of roles. See [DROP ROLE](DROP-ROLE.html) for more information.
+
+## Parameters<a id="topic1__section4"></a>
+
+IF EXISTS
+Do not throw an error if the role does not exist. A notice is issued in this case.
+
+ *name*
+The name of an existing role.
+
+## Compatibility<a id="topic1__section5"></a>
+
+There is no `DROP GROUP` statement in the SQL standard.
+
+## See Also<a id="topic1__section6"></a>
+
+[DROP ROLE](DROP-ROLE.html)
diff --git a/reference/sql/DROP-OPERATOR-CLASS.html.md.erb b/reference/sql/DROP-OPERATOR-CLASS.html.md.erb
new file mode 100644
index 0000000..cc1050d
--- /dev/null
+++ b/reference/sql/DROP-OPERATOR-CLASS.html.md.erb
@@ -0,0 +1,54 @@
+---
+title: DROP OPERATOR CLASS
+---
+
+Removes an operator class.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+DROP OPERATOR CLASS [IF EXISTS] name USING index_method [CASCADE | RESTRICT]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`DROP OPERATOR` drops an existing operator class. To execute this command you must be the owner of the operator class.
+
+## Parameters<a id="topic1__section4"></a>
+
+IF EXISTS
+Do not throw an error if the operator class does not exist. A notice is issued in this case.
+
+ *name*
+The name (optionally schema-qualified) of an existing operator class.
+
+ *index\_method*
+The name of the index access method the operator class is for.
+
+CASCADE
+Automatically drop objects that depend on the operator class.
+
+RESTRICT
+Refuse to drop the operator class if any objects depend on it. This is the default.
+
+## Notes
+
+This command will not succeed if there are any existing indexes that use the operator class. Add `CASCADE` to drop such indexes along with the operator class.
+
+## Examples<a id="topic1__section5"></a>
+
+Remove the B-tree operator class `widget_ops`:
+
+``` pre
+DROP OPERATOR CLASS widget_ops USING btree;
+```
+
+This command will not succeed if there are any existing indexes that use the operator class. Add `CASCADE` to drop such indexes along with the operator class.
+
+## Compatibility<a id="topic1__section6"></a>
+
+There is no `DROP OPERATOR CLASS` statement in the SQL standard.
+
+## See Also<a id="topic1__section7"></a>
+
+[ALTER OPERATOR](ALTER-OPERATOR.html), [CREATE OPERATOR](CREATE-OPERATOR.html) [CREATE OPERATOR CLASS](CREATE-OPERATOR-CLASS.html)
diff --git a/reference/sql/DROP-OPERATOR.html.md.erb b/reference/sql/DROP-OPERATOR.html.md.erb
new file mode 100644
index 0000000..7135b26
--- /dev/null
+++ b/reference/sql/DROP-OPERATOR.html.md.erb
@@ -0,0 +1,64 @@
+---
+title: DROP OPERATOR
+---
+
+Removes an operator.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+DROP OPERATOR [IF EXISTS] name ( {lefttype | NONE} ,
+ {righttype | NONE} ) [CASCADE | RESTRICT]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`DROP OPERATOR` drops an existing operator from the database system. To execute this command you must be the owner of the operator.
+
+## Parameters<a id="topic1__section4"></a>
+
+IF EXISTS
+Do not throw an error if the operator does not exist. A notice is issued in this case.
+
+ *name*
+The name (optionally schema-qualified) of an existing operator.
+
+ *lefttype*
+The data type of the operator's left operand; write `NONE` if the operator has no left operand.
+
+ *righttype*
+The data type of the operator's right operand; write `NONE` if the operator has no right operand.
+
+CASCADE
+Automatically drop objects that depend on the operator.
+
+RESTRICT
+Refuse to drop the operator if any objects depend on it. This is the default.
+
+## Examples<a id="topic1__section5"></a>
+
+Remove the power operator `a^b` for type `integer`:
+
+``` pre
+DROP OPERATOR ^ (integer, integer);
+```
+
+Remove the left unary bitwise complement operator `~b` for type `bit`:
+
+``` pre
+DROP OPERATOR ~ (none, bit);
+```
+
+Remove the right unary factorial operator `x!` for type `bigint`:
+
+``` pre
+DROP OPERATOR ! (bigint, none);
+```
+
+## Compatibility<a id="topic1__section6"></a>
+
+There is no `DROP OPERATOR` statement in the SQL standard.
+
+## See Also<a id="topic1__section7"></a>
+
+[ALTER OPERATOR](ALTER-OPERATOR.html), [CREATE OPERATOR](CREATE-OPERATOR.html)
diff --git a/reference/sql/DROP-OWNED.html.md.erb b/reference/sql/DROP-OWNED.html.md.erb
new file mode 100644
index 0000000..920727d
--- /dev/null
+++ b/reference/sql/DROP-OWNED.html.md.erb
@@ -0,0 +1,50 @@
+---
+title: DROP OWNED
+---
+
+Removes database objects owned by a database role.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+DROP OWNED BY name [, ...] [CASCADE | RESTRICT]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`DROP OWNED` drops all the objects in the current database that are owned by one of the specified roles. Any privileges granted to the given roles on objects in the current database will also be revoked.
+
+## Parameters<a id="topic1__section4"></a>
+
+ *name*
+The name of a role whose objects will be dropped, and whose privileges will be revoked.
+
+CASCADE
+Automatically drop objects that depend on the affected objects.
+
+RESTRICT
+Refuse to drop the objects owned by a role if any other database objects depend on one of the affected objects. This is the default.
+
+## Notes<a id="topic1__section5"></a>
+
+`DROP OWNED` is often used to prepare for the removal of one or more roles. Because `DROP OWNED` only affects the objects in the current database, it is usually necessary to execute this command in each database that contains objects owned by a role that is to be removed.
+
+Using the `CASCADE` option may make the command recurse to objects owned by other users.
+
+The `REASSIGN OWNED` command is an alternative that reassigns the ownership of all the database objects owned by one or more roles.
+
+## Examples<a id="topic1__section6"></a>
+
+Remove any database objects owned by the role named `sally`:
+
+``` pre
+DROP OWNED BY sally;
+```
+
+## Compatibility<a id="topic1__section7"></a>
+
+The `DROP OWNED` statement is a HAWQ extension.
+
+## See Also<a id="topic1__section8"></a>
+
+[REASSIGN OWNED](REASSIGN-OWNED.html), [DROP ROLE](DROP-ROLE.html)
diff --git a/reference/sql/DROP-RESOURCE-QUEUE.html.md.erb b/reference/sql/DROP-RESOURCE-QUEUE.html.md.erb
new file mode 100644
index 0000000..933e147
--- /dev/null
+++ b/reference/sql/DROP-RESOURCE-QUEUE.html.md.erb
@@ -0,0 +1,65 @@
+---
+title: DROP RESOURCE QUEUE
+---
+
+Removes a resource queue.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+DROP RESOURCE QUEUE queue_name
+
+```
+
+## Description<a id="topic1__section3"></a>
+
+This command removes a resource queue from HAWQ. To drop a resource queue, the queue cannot have any roles assigned to it, nor can it have any statements waiting in the queue or have any children resource queues. Only a superuser can drop a resource queue.
+
+**Note:** The `pg_root` and `pg_default` resource queues cannot be dropped.
+
+## Parameters<a id="topic1__section4"></a>
+
+ *queue\_name*
+The name of a resource queue to remove.
+
+## Notes<a id="topic1__section5"></a>
+
+Use [ALTER ROLE](ALTER-ROLE.html) to remove a user from a resource queue.
+
+To see all the currently active queries for all resource queues, perform the following query of the `pg_locks` table joined with the `pg_roles` and `pg_resqueue` tables:
+
+``` pre
+SELECT rolname, rsqname, locktype, objid, transaction, pid,
+mode, granted FROM pg_roles, pg_resqueue, pg_locks WHERE
+pg_roles.rolresqueue=pg_locks.objid AND
+pg_locks.objid=pg_resqueue.oid;
+```
+
+To see the roles assigned to a resource queue, perform the following query of the `pg_roles` and `pg_resqueue` system catalog tables:
+
+``` pre
+SELECT rolname, rsqname FROM pg_roles, pg_resqueue WHERE
+pg_roles.rolresqueue=pg_resqueue.oid;
+```
+
+## Examples<a id="topic1__section6"></a>
+
+Remove a role from a resource queue (and move the role to the default resource queue, `pg_default`):
+
+``` pre
+ALTER ROLE bob RESOURCE QUEUE NONE;
+```
+
+Remove the resource queue named `adhoc`:
+
+``` pre
+DROP RESOURCE QUEUE adhoc;
+```
+
+## Compatibility<a id="topic1__section7"></a>
+
+The `DROP RESOURCE QUEUE` statement is a HAWQ extension.
+
+## See Also<a id="topic1__section8"></a>
+
+[CREATE RESOURCE QUEUE](CREATE-RESOURCE-QUEUE.html), [ALTER ROLE](ALTER-ROLE.html), [ALTER RESOURCE QUEUE](ALTER-RESOURCE-QUEUE.html)
diff --git a/reference/sql/DROP-ROLE.html.md.erb b/reference/sql/DROP-ROLE.html.md.erb
new file mode 100644
index 0000000..ce25582
--- /dev/null
+++ b/reference/sql/DROP-ROLE.html.md.erb
@@ -0,0 +1,43 @@
+---
+title: DROP ROLE
+---
+
+Removes a database role.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+DROP ROLE [IF EXISTS] name [, ...]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`DROP ROLE` removes the specified role(s). To drop a superuser role, you must be a superuser yourself. To drop non-superuser roles, you must have `CREATEROLE` privilege.
+
+A role cannot be removed if it is still referenced in any database; an error will be raised if so. Before dropping the role, you must drop all the objects it owns (or reassign their ownership) and revoke any privileges the role has been granted. The `REASSIGN OWNED` and `DROP OWNED` commands can be useful for this purpose.
+
+However, it is not necessary to remove role memberships involving the role; `DROP ROLE` automatically revokes any memberships of the target role in other roles, and of other roles in the target role. The other roles are not dropped nor otherwise affected.
+
+## Parameters<a id="topic1__section4"></a>
+
+IF EXISTS
+Do not throw an error if the role does not exist. A notice is issued in this case.
+
+ *name*
+The name of the role to remove.
+
+## Examples<a id="topic1__section5"></a>
+
+Remove the roles named `sally` and `bob`:
+
+``` pre
+DROP ROLE sally, bob;
+```
+
+## Compatibility<a id="topic1__section6"></a>
+
+The SQL standard defines `DROP ROLE`, but it allows only one role to be dropped at a time, and it specifies different privilege requirements than HAWQ uses.
+
+## See Also<a id="topic1__section7"></a>
+
+[ALTER ROLE](ALTER-ROLE.html), [CREATE ROLE](CREATE-ROLE.html), [DROP OWNED](DROP-OWNED.html), [REASSIGN OWNED](REASSIGN-OWNED.html), [SET ROLE](SET-ROLE.html)
diff --git a/reference/sql/DROP-SCHEMA.html.md.erb b/reference/sql/DROP-SCHEMA.html.md.erb
new file mode 100644
index 0000000..5f5cd41
--- /dev/null
+++ b/reference/sql/DROP-SCHEMA.html.md.erb
@@ -0,0 +1,45 @@
+---
+title: DROP SCHEMA
+---
+
+Removes a schema.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+DROP SCHEMA [IF EXISTS] name [, ...] [CASCADE | RESTRICT]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`DROP SCHEMA` removes schemas from the database. A schema can only be dropped by its owner or a superuser. Note that the owner can drop the schema (and thereby all contained objects) even if he does not own some of the objects within the schema.
+
+## Parameters<a id="topic1__section4"></a>
+
+IF EXISTS
+Do not throw an error if the schema does not exist. A notice is issued in this case.
+
+ *name*
+The name of the schema to remove.
+
+CASCADE
+Automatically drops any objects contained in the schema (tables, functions, etc.).
+
+RESTRICT
+Refuse to drop the schema if it contains any objects. This is the default.
+
+## Examples<a id="topic1__section5"></a>
+
+Remove the schema `mystuff` from the database, along with everything it contains:
+
+``` pre
+DROP SCHEMA mystuff CASCADE;
+```
+
+## Compatibility<a id="topic1__section6"></a>
+
+`DROP SCHEMA` is fully conforming with the SQL standard, except that the standard only allows one schema to be dropped per command. Also, the `IF EXISTS` option is a HAWQ extension.
+
+## See Also<a id="topic1__section7"></a>
+
+[CREATE SCHEMA](CREATE-SCHEMA.html)
diff --git a/reference/sql/DROP-SEQUENCE.html.md.erb b/reference/sql/DROP-SEQUENCE.html.md.erb
new file mode 100644
index 0000000..5cbc5fe
--- /dev/null
+++ b/reference/sql/DROP-SEQUENCE.html.md.erb
@@ -0,0 +1,45 @@
+---
+title: DROP SEQUENCE
+---
+
+Removes a sequence.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+DROP SEQUENCE [IF EXISTS] name [, ...] [CASCADE | RESTRICT]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`DROP SEQUENCE` removes a sequence generator table. You must own the sequence to drop it (or be a superuser).
+
+## Parameters<a id="topic1__section4"></a>
+
+IF EXISTS
+Do not throw an error if the sequence does not exist. A notice is issued in this case.
+
+ *name*
+The name (optionally schema-qualified) of the sequence to remove.
+
+CASCADE
+Automatically drop objects that depend on the sequence.
+
+RESTRICT
+Refuse to drop the sequence if any objects depend on it. This is the default.
+
+## Examples<a id="topic1__section5"></a>
+
+Remove the sequence `myserial`:
+
+``` pre
+DROP SEQUENCE myserial;
+```
+
+## Compatibility<a id="topic1__section6"></a>
+
+`DROP SEQUENCE` is fully conforming with the SQL standard, except that the standard only allows one sequence to be dropped per command. Also, the `IF EXISTS` option is a HAWQ extension.
+
+## See Also<a id="topic1__section7"></a>
+
+[CREATE SEQUENCE](CREATE-SEQUENCE.html)
diff --git a/reference/sql/DROP-TABLE.html.md.erb b/reference/sql/DROP-TABLE.html.md.erb
new file mode 100644
index 0000000..ec0eecb
--- /dev/null
+++ b/reference/sql/DROP-TABLE.html.md.erb
@@ -0,0 +1,47 @@
+---
+title: DROP TABLE
+---
+
+Removes a table.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+DROP TABLE [IF EXISTS] name [, ...] [CASCADE | RESTRICT]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`DROP TABLE` removes tables from the database. Only its owner may drop a table. To empty a table of rows without removing the table definition, use `DELETE` or `TRUNCATE`.
+
+`DROP TABLE` always removes any indexes, rules, and constraints that exist for the target table. However, to drop a table that is referenced by a view, `CASCADE` must be specified. `CASCADE` will remove a dependent view entirely.
+
+## Parameters<a id="topic1__section4"></a>
+
+IF EXISTS
+Do not throw an error if the table does not exist. A notice is issued in this case.
+
+ *name*
+The name (optionally schema-qualified) of the table to remove.
+
+CASCADE
+Automatically drop objects that depend on the table (such as views).
+
+RESTRICT
+Refuse to drop the table if any objects depend on it. This is the default.
+
+## Examples<a id="topic1__section5"></a>
+
+Remove the table `mytable`:
+
+``` pre
+DROP TABLE mytable;
+```
+
+## Compatibility<a id="topic1__section6"></a>
+
+`DROP TABLE` is fully conforming with the SQL standard, except that the standard only allows one table to be dropped per command. Also, the `IF EXISTS` option is a HAWQ extension.
+
+## See Also<a id="topic1__section7"></a>
+
+[CREATE TABLE](CREATE-TABLE.html), [ALTER TABLE](ALTER-TABLE.html), [TRUNCATE](TRUNCATE.html)
diff --git a/reference/sql/DROP-TABLESPACE.html.md.erb b/reference/sql/DROP-TABLESPACE.html.md.erb
new file mode 100644
index 0000000..61aa8b6
--- /dev/null
+++ b/reference/sql/DROP-TABLESPACE.html.md.erb
@@ -0,0 +1,42 @@
+---
+title: DROP TABLESPACE
+---
+
+Removes a tablespace.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+DROP TABLESPACE [IF EXISTS] tablespacename
+
+```
+
+## Description<a id="topic1__section3"></a>
+
+`DROP TABLESPACE` removes a tablespace from the system.
+
+A tablespace can only be dropped by its owner or a superuser. The tablespace must be empty of all database objects before it can be dropped. It is possible that objects in other databases may still reside in the tablespace even if no objects in the current database are using the tablespace.
+
+## Parameters<a id="topic1__section4"></a>
+
+IF EXISTS
+Do not throw an error if the tablespace does not exist. A notice is issued in this case.
+
+ *tablespacename*
+The name of the tablespace to remove.
+
+## Examples<a id="topic1__section5"></a>
+
+Remove the tablespace `mystuff`:
+
+``` pre
+DROP TABLESPACE mystuff;
+```
+
+## Compatibility<a id="topic1__section6"></a>
+
+`DROP TABLESPACE` is a HAWQ extension.
+
+## See Also<a id="topic1__section7"></a>
+
+[CREATE TABLESPACE](CREATE-TABLESPACE.html), [ALTER TABLESPACE](ALTER-TABLESPACE.html)
diff --git a/reference/sql/DROP-TYPE.html.md.erb b/reference/sql/DROP-TYPE.html.md.erb
new file mode 100644
index 0000000..204dd23
--- /dev/null
+++ b/reference/sql/DROP-TYPE.html.md.erb
@@ -0,0 +1,45 @@
+---
+title: DROP TYPE
+---
+
+Removes a data type.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+DROP TYPE [IF EXISTS] name [, ...] [CASCADE | RESTRICT]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`DROP TYPE` will remove a user-defined data type. Only the owner of a type can remove it.
+
+## Parameters<a id="topic1__section4"></a>
+
+IF EXISTS
+Do not throw an error if the type does not exist. A notice is issued in this case.
+
+ *name*
+The name (optionally schema-qualified) of the data type to remove.
+
+CASCADE
+Automatically drop objects that depend on the type (such as table columns, functions, operators).
+
+RESTRICT
+Refuse to drop the type if any objects depend on it. This is the default.
+
+## Examples<a id="topic1__section5"></a>
+
+Remove the data type `box`;
+
+``` pre
+DROP TYPE box;
+```
+
+## Compatibility<a id="topic1__section6"></a>
+
+This command is similar to the corresponding command in the SQL standard, apart from the `IF EXISTS` option, which is a HAWQ extension. But note that the `CREATE TYPE` command and the data type extension mechanisms in HAWQ differ from the SQL standard.
+
+## See Also<a id="topic1__section7"></a>
+
+[ALTER TYPE](ALTER-TYPE.html), [CREATE TYPE](CREATE-TYPE.html)
diff --git a/reference/sql/DROP-USER.html.md.erb b/reference/sql/DROP-USER.html.md.erb
new file mode 100644
index 0000000..51d0782
--- /dev/null
+++ b/reference/sql/DROP-USER.html.md.erb
@@ -0,0 +1,31 @@
+---
+title: DROP USER
+---
+
+Removes a database role.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+DROP USER [IF EXISTS] name [, ...]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`DROP USER` is an obsolete command, though still accepted for backwards compatibility. Groups (and users) have been superseded by the more general concept of roles. See [DROP ROLE](DROP-ROLE.html) for more information.
+
+## Parameters<a id="topic1__section4"></a>
+
+IF EXISTS
+Do not throw an error if the role does not exist. A notice is issued in this case.
+
+ *name*
+The name of an existing role.
+
+## Compatibility<a id="topic1__section5"></a>
+
+There is no `DROP USER` statement in the SQL standard. The SQL standard leaves the definition of users to the implementation.
+
+## See Also<a id="topic1__section6"></a>
+
+[DROP ROLE](DROP-ROLE.html)
diff --git a/reference/sql/DROP-VIEW.html.md.erb b/reference/sql/DROP-VIEW.html.md.erb
new file mode 100644
index 0000000..8dea57c
--- /dev/null
+++ b/reference/sql/DROP-VIEW.html.md.erb
@@ -0,0 +1,45 @@
+---
+title: DROP VIEW
+---
+
+Removes a view.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+DROP VIEW [IF EXISTS] name [, ...] [CASCADE | RESTRICT]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`DROP VIEW` will remove an existing view. Only the owner of a view can remove it.
+
+## Parameters<a id="topic1__section4"></a>
+
+IF EXISTS
+Do not throw an error if the view does not exist. A notice is issued in this case.
+
+ *name*
+The name (optionally schema-qualified) of the view to remove.
+
+CASCADE
+Automatically drop objects that depend on the view (such as other views).
+
+RESTRICT
+Refuse to drop the view if any objects depend on it. This is the default.
+
+## Examples<a id="topic1__section5"></a>
+
+Remove the view `topten`;
+
+``` pre
+DROP VIEW topten;
+```
+
+## Compatibility<a id="topic1__section6"></a>
+
+`DROP VIEW` is fully conforming with the SQL standard, except that the standard only allows one view to be dropped per command. Also, the `IF EXISTS` option is a HAWQ extension.
+
+## See Also<a id="topic1__section7"></a>
+
+[CREATE VIEW](CREATE-VIEW.html)
diff --git a/reference/sql/END.html.md.erb b/reference/sql/END.html.md.erb
new file mode 100644
index 0000000..ffa68a6
--- /dev/null
+++ b/reference/sql/END.html.md.erb
@@ -0,0 +1,37 @@
+---
+title: END
+---
+
+Commits the current transaction.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+END [WORK | TRANSACTION]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`END` commits the current transaction. All changes made by the transaction become visible to others and are guaranteed to be durable if a crash occurs. This command is a HAWQ extension that is equivalent to [COMMIT](COMMIT.html).
+
+## Parameters<a id="topic1__section4"></a>
+
+WORK
+TRANSACTION
+Optional keywords. They have no effect.
+
+## Examples<a id="topic1__section5"></a>
+
+Commit the current transaction:
+
+``` pre
+END;
+```
+
+## Compatibility<a id="topic1__section6"></a>
+
+`END` is a HAWQ extension that provides functionality equivalent to [COMMIT](COMMIT.html), which is specified in the SQL standard.
+
+## See Also<a id="topic1__section7"></a>
+
+[BEGIN](BEGIN.html), [ROLLBACK](ROLLBACK.html), [COMMIT](COMMIT.html)
diff --git a/reference/sql/EXECUTE.html.md.erb b/reference/sql/EXECUTE.html.md.erb
new file mode 100644
index 0000000..9212498
--- /dev/null
+++ b/reference/sql/EXECUTE.html.md.erb
@@ -0,0 +1,45 @@
+---
+title: EXECUTE
+---
+
+Executes a prepared SQL statement.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+EXECUTE name [ (parameter [, ...] ) ]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`EXECUTE` is used to execute a previously prepared statement. Since prepared statements only exist for the duration of a session, the prepared statement must have been created by a `PREPARE` statement executed earlier in the current session.
+
+If the `PREPARE` statement that created the statement specified some parameters, a compatible set of parameters must be passed to the `EXECUTE` statement, or else an error is raised. Note that (unlike functions) prepared statements are not overloaded based on the type or number of their parameters; the name of a prepared statement must be unique within a database session.
+
+For more information on the creation and usage of prepared statements, see `PREPARE`.
+
+## Parameters<a id="topic1__section4"></a>
+
+ *name*
+The name of the prepared statement to execute.
+
+ *parameter*
+The actual value of a parameter to the prepared statement. This must be an expression yielding a value that is compatible with the data type of this parameter, as was determined when the prepared statement was created.
+
+## Examples<a id="topic1__section5"></a>
+
+Create a prepared statement for an `INSERT` statement, and then execute it:
+
+``` pre
+PREPARE fooplan (int, text, bool, numeric) AS INSERT INTO
+foo VALUES($1, $2, $3, $4);
+EXECUTE fooplan(1, 'Hunter Valley', 't', 200.00);
+```
+
+## Compatibility<a id="topic1__section6"></a>
+
+The SQL standard includes an `EXECUTE` statement, but it is only for use in embedded SQL. This version of the `EXECUTE` statement also uses a somewhat different syntax.
+
+## See Also<a id="topic1__section7"></a>
+
+[DEALLOCATE](DEALLOCATE.html), [PREPARE](PREPARE.html)
diff --git a/reference/sql/EXPLAIN.html.md.erb b/reference/sql/EXPLAIN.html.md.erb
new file mode 100644
index 0000000..e916bbe
--- /dev/null
+++ b/reference/sql/EXPLAIN.html.md.erb
@@ -0,0 +1,94 @@
+---
+title: EXPLAIN
+---
+
+Shows the query plan of a statement.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+EXPLAIN [ANALYZE] [VERBOSE] statement
+
+```
+
+## Description<a id="topic1__section3"></a>
+
+`EXPLAIN` displays the query plan that the HAWQ planner generates for the supplied statement. Query plans are a tree plan of nodes. Each node in the plan represents a single operation, such as table scan, join, aggregation or a sort.
+
+Plans should be read from the bottom up as each node feeds rows into the node directly above it. The bottom nodes of a plan are usually table scan operations. If the query requires joins, aggregations, or sorts (or other operations on the raw rows), then there will be additional nodes above the scan nodes to perform these operations. The topmost plan nodes are usually the HAWQ motion nodes (redistribute, explicit redistribute, broadcast, or gather motions). These are the operations responsible for moving rows between the segment instances during query processing.
+
+The output of `EXPLAIN` has one line for each node in the plan tree, showing the basic node type plus the following cost estimates that the planner made for the execution of that plan node:
+
+- **cost** — measured in units of disk page fetches; that is, 1.0 equals one sequential disk page read. The first estimate is the start-up cost (cost of getting to the first row) and the second is the total cost (cost of getting all rows). Note that the total cost assumes that all rows will be retrieved, which may not always be the case (if using `LIMIT` for example).
+- **rows** — the total number of rows output by this plan node. This is usually less than the actual number of rows processed or scanned by the plan node, reflecting the estimated selectivity of any `WHERE` clause conditions. Ideally the top-level nodes estimate will approximate the number of rows actually returned, updated, or deleted by the query.
+- **width** — total bytes of all the rows output by this plan node.
+
+It is important to note that the cost of an upper-level node includes the cost of all its child nodes. The topmost node of the plan has the estimated total execution cost for the plan. This is this number that the planner seeks to minimize. It is also important to realize that the cost only reflects things that the query planner cares about. In particular, the cost does not consider the time spent transmitting result rows to the client.
+
+`EXPLAIN ANALYZE` causes the statement to be actually executed, not only planned. The `EXPLAIN ANALYZE` plan shows the actual results along with the planner's estimates. This is useful for seeing whether the planner's estimates are close to reality. In addition to the information shown in the `EXPLAIN` plan, `EXPLAIN ANALYZE` will show the following additional information:
+
+- The total elapsed time (in milliseconds) that it took to run the query.
+- The number of *workers* (segments) involved in a plan node operation. Only segments that return rows are counted.
+- The maximum number of rows returned by the segment that produced the most rows for an operation. If multiple segments produce an equal number of rows, the one with the longest *time to end* is the one chosen.
+- The segment id number of the segment that produced the most rows for an operation.
+- For relevant operations, the *work\_mem* used by the operation. If *work\_mem* was not sufficient to perform the operation in memory, the plan will show how much data was spilled to disk and how many passes over the data were required for the lowest performing segment. For example:
+
+ ``` pre
+ Work_mem used: 64K bytes avg, 64K bytes max (seg0).
+ Work_mem wanted: 90K bytes avg, 90K bytes max (seg0) to abate workfile
+ I/O affecting 2 workers.
+ [seg0] pass 0: 488 groups made from 488 rows; 263 rows written to
+ workfile
+ [seg0] pass 1: 263 groups made from 263 rows
+ ```
+
+- The time (in milliseconds) it took to retrieve the first row from the segment that produced the most rows, and the total time taken to retrieve all rows from that segment. The *<time> to first row* may be omitted if it is the same as the *<time> to end*.
+
+**Important:**
+Keep in mind that the statement is actually executed when `EXPLAIN ANALYZE` is used. Although `EXPLAIN ANALYZE` will discard any output that a `SELECT` would return, other side effects of the statement will happen as usual. If you wish to use `EXPLAIN ANALYZE` on a DML statement without letting the command affect your data, use this approach:
+
+``` pre
+BEGIN;
+EXPLAIN ANALYZE ...;
+ROLLBACK;
+```
+
+## Parameters<a id="topic1__section4"></a>
+
+ *name*
+The name of the prepared statement to execute.
+
+ *parameter*
+The actual value of a parameter to the prepared statement. This must be an expression yielding a value that is compatible with the data type of this parameter, as was determined when the prepared statement was created.
+
+## Notes<a id="topic1__section5"></a>
+
+In order to allow the query planner to make reasonably informed decisions when optimizing queries, the `ANALYZE` statement should be run to record statistics about the distribution of data within the table. If you have not done this (or if the statistical distribution of the data in the table has changed significantly since the last time `ANALYZE` was run), the estimated costs are unlikely to conform to the real properties of the query, and consequently an inferior query plan may be chosen.
+
+## Examples<a id="topic1__section6"></a>
+
+To illustrate how to read an `EXPLAIN` query plan, consider the following example for a very simple query:
+
+``` pre
+EXPLAIN SELECT * FROM names WHERE name = 'Joelle';
+ QUERY PLAN
+------------------------------------------------------------
+Gather Motion 2:1 (slice1) (cost=0.00..20.88 rows=1 width=13)
+
+ -> Seq Scan on 'names' (cost=0.00..20.88 rows=1 width=13)
+ Filter: name::text ~~ 'Joelle'::text
+```
+
+If we read the plan from the bottom up, the query planner starts by doing a sequential scan of the `names` table. Notice that the `WHERE` clause is being applied as a *filter* condition. This means that the scan operation checks the condition for each row it scans, and outputs only the ones that pass the condition.
+
+The results of the scan operation are passed up to a *gather motion* operation. In HAWQ, a gather motion is when segments send rows up to the master. In this case we have 2 segment instances sending to 1 master instance (2:1). This operation is working on `slice1` of the parallel query execution plan. In HAWQ, a query plan is divided into *slices* so that portions of the query plan can be worked on in parallel by the segments.
+
+The estimated startup cost for this plan is `00.00` (no cost) and a total cost of `20.88` disk page fetches. The planner is estimating that this query will return one row.
+
+## Compatibility<a id="topic1__section7"></a>
+
+There is no `EXPLAIN` statement defined in the SQL standard.
+
+## See Also<a id="topic1__section8"></a>
+
+[ANALYZE](ANALYZE.html)
diff --git a/reference/sql/FETCH.html.md.erb b/reference/sql/FETCH.html.md.erb
new file mode 100644
index 0000000..28ca399
--- /dev/null
+++ b/reference/sql/FETCH.html.md.erb
@@ -0,0 +1,146 @@
+---
+title: FETCH
+---
+
+Retrieves rows from a query using a cursor.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+FETCH [ forward_direction { FROM | IN } ] cursorname
+
+```
+
+where *forward\_direction* can be empty or one of:
+
+``` pre
+ NEXT
+ FIRST
+ LAST
+ ABSOLUTE count
+ RELATIVE count
+ count
+ ALL
+ FORWARD
+ FORWARD count
+ FORWARD ALL
+```
+
+## Description<a id="topic1__section3"></a>
+
+`FETCH` retrieves rows using a previously-created cursor.
+
+A cursor has an associated position, which is used by `FETCH`. The cursor position can be before the first row of the query result, on any particular row of the result, or after the last row of the result. When created, a cursor is positioned before the first row. After fetching some rows, the cursor is positioned on the row most recently retrieved. If `FETCH` runs off the end of the available rows then the cursor is left positioned after the last row. `FETCH ALL` will always leave the cursor positioned after the last row.
+
+The forms `NEXT`, `FIRST`, `LAST`, `ABSOLUTE`, `RELATIVE` fetch a single row after moving the cursor appropriately. If there is no such row, an empty result is returned, and the cursor is left positioned before the first row or after the last row as appropriate.
+
+The forms using `FORWARD` retrieve the indicated number of rows moving in the forward direction, leaving the cursor positioned on the last-returned row (or after all rows, if the count exceeds the number of rows available). Note that it is not possible to move a cursor position backwards in HAWQ, since scrollable cursors are not supported. You can only move a cursor forward in position using `FETCH`.
+
+`RELATIVE 0` and `FORWARD 0` request fetching the current row without moving the cursor, that is, re-fetching the most recently fetched row. This will succeed unless the cursor is positioned before the first row or after the last row, in which case no row is returned.
+
+**Outputs**
+
+On successful completion, a `FETCH` command returns a command tag of the form
+
+``` pre
+FETCH count
+
+```
+
+The count is the number of rows fetched (possibly zero). Note that in `psql`, the command tag will not actually be displayed, since `psql` displays the fetched rows instead.
+
+## Parameters<a id="topic1__section5"></a>
+
+ *forward\_direction*
+Defines the fetch direction and number of rows to fetch. Only forward fetches are allowed in HAWQ. It can be one of the following:
+
+NEXT
+Fetch the next row. This is the default if direction is omitted.
+
+FIRST
+Fetch the first row of the query (same as `ABSOLUTE 1`). Only allowed if it is the first `FETCH` operation using this cursor.
+
+LAST
+Fetch the last row of the query (same as `ABSOLUTE -1`).
+
+ABSOLUTE *count*
+Fetch the specified row of the query. Position after last row if count is out of range. Only allowed if the row specified by *count* moves the cursor position forward.
+
+RELATIVE *count*
+Fetch the specified row of the query *count* rows ahead of the current cursor position. `RELATIVE 0` re-fetches the current row, if any. Only allowed if *count* moves the cursor position forward.
+
+ *count*
+Fetch the next *count* number of rows (same as `FORWARD count `).
+
+ALL
+Fetch all remaining rows (same as `FORWARD ALL`).
+
+FORWARD
+Fetch the next row (same as `NEXT`).
+
+FORWARD *count*
+Fetch the next *count* number of rows. `FORWARD 0` re-fetches the current row.
+
+FORWARD ALL
+Fetch all remaining rows.
+
+ *cursorname*
+The name of an open cursor.
+
+## Notes<a id="topic1__section6"></a>
+
+HAWQ does not support scrollable cursors, so you can only use `FETCH` to move the cursor position forward.
+
+`ABSOLUTE` fetches are not any faster than navigating to the desired row with a relative move: the underlying implementation must traverse all the intermediate rows anyway.
+
+Updating data via a cursor is currently not supported by HAWQ.
+
+`DECLARE` is used to define a cursor. Use `MOVE` to change cursor position without retrieving data.
+
+## Examples<a id="topic1__section7"></a>
+
+-- Start the transaction:
+
+``` pre
+BEGIN;
+```
+
+-- Set up a cursor:
+
+``` pre
+DECLARE mycursor CURSOR FOR SELECT * FROM films;
+```
+
+-- Fetch the first 5 rows in the cursor `mycursor`:
+
+``` pre
+FETCH FORWARD 5 FROM mycursor;
+ code | title | did | date_prod | kind | len
+-------+-------------------------+-----+------------+----------+-------
+ BL101 | The Third Man | 101 | 1949-12-23 | Drama | 01:44
+ BL102 | The African Queen | 101 | 1951-08-11 | Romantic | 01:43
+ JL201 | Une Femme est une Femme | 102 | 1961-03-12 | Romantic | 01:25
+ P_301 | Vertigo | 103 | 1958-11-14 | Action | 02:08
+ P_302 | Becket | 103 | 1964-02-03 | Drama | 02:28
+```
+
+-- Close the cursor and end the transaction:
+
+``` pre
+CLOSE mycursor;
+COMMIT;
+```
+
+## Compatibility<a id="topic1__section8"></a>
+
+SQL standard allows cursors only in embedded SQL and in modules. HAWQ permits cursors to be used interactively.
+
+The variant of `FETCH` described here returns the data as if it were a `SELECT` result rather than placing it in host variables. Other than this point, `FETCH` is fully upward-compatible with the SQL standard.
+
+The `FETCH` forms involving `FORWARD`, as well as the forms `FETCH` count and `FETCH ALL`, in which `FORWARD` is implicit, are HAWQ extensions. `BACKWARD` is not supported.
+
+The SQL standard allows only `FROM` preceding the cursor name; the option to use `IN` is an extension.
+
+## See Also<a id="topic1__section9"></a>
+
+[DECLARE](DECLARE.html), [CLOSE](CLOSE.html)
diff --git a/reference/sql/GRANT.html.md.erb b/reference/sql/GRANT.html.md.erb
new file mode 100644
index 0000000..55c0e2e
--- /dev/null
+++ b/reference/sql/GRANT.html.md.erb
@@ -0,0 +1,180 @@
+---
+title: GRANT
+---
+
+Defines access privileges.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+GRANT { {SELECT | INSERT | UPDATE | DELETE | REFERENCES }
+[,...] | ALL [PRIVILEGES] }
+ ON [TABLE] tablename [, ...]
+ TO {rolename | PUBLIC} [, ...] [WITH GRANT OPTION]
+
+GRANT { {USAGE | SELECT | UPDATE} [,...] | ALL [PRIVILEGES] }
+ ON SEQUENCE sequencename [, ...]
+ TO { rolename | PUBLIC } [, ...] [WITH GRANT OPTION]
+
+GRANT { {CREATE | CONNECT | TEMPORARY | TEMP} [,...] | ALL
+[PRIVILEGES] }
+ ON DATABASE dbname [, ...]
+ TO {rolename | PUBLIC} [, ...] [WITH GRANT OPTION]
+
+GRANT { EXECUTE | ALL [PRIVILEGES] }
+ ON FUNCTION funcname ( [ [argmode] [argname] argtype [, ...]
+ ] ) [, ...]
+ TO {rolename | PUBLIC} [, ...] [WITH GRANT OPTION]
+
+GRANT { USAGE | ALL [PRIVILEGES] }
+ ON LANGUAGE langname [, ...]
+ TO {rolename | PUBLIC} [, ...] [WITH GRANT OPTION]
+
+GRANT { {CREATE | USAGE} [,...] | ALL [PRIVILEGES] }
+ ON SCHEMA schemaname [, ...]
+ TO {rolename | PUBLIC} [, ...] [WITH GRANT OPTION]
+
+GRANT { CREATE | ALL [PRIVILEGES] }
+ ON TABLESPACE tablespacename [, ...]
+ TO {rolename | PUBLIC} [, ...] [WITH GRANT OPTION]
+
+GRANT parent_role [, ...]
+ TO member_role [, ...] [WITH ADMIN OPTION]
+
+GRANT { SELECT | INSERT | ALL [PRIVILEGES] }
+ ON PROTOCOL protocolname
+ TO username
+
+```
+
+## Description<a id="topic1__section3"></a>
+
+The `GRANT` command has two basic variants: one that grants privileges on a database object (table, view, sequence, database, function, procedural language, schema, or tablespace), and one that grants membership in a role.
+
+**GRANT on Database Objects**
+
+This variant of the `GRANT` command gives specific privileges on a database object to one or more roles. These privileges are added to those already granted, if any.
+
+The key word `PUBLIC` indicates that the privileges are to be granted to all roles, including those that may be created later. `PUBLIC` may be thought of as an implicitly defined group-level role that always includes all roles. Any particular role will have the sum of privileges granted directly to it, privileges granted to any role it is presently a member of, and privileges granted to `PUBLIC`.
+
+If `WITH GRANT OPTION` is specified, the recipient of the privilege may in turn grant it to others. Without a grant option, the recipient cannot do that. Grant options cannot be granted to `PUBLIC`.
+
+There is no need to grant privileges to the owner of an object (usually the role that created it), as the owner has all privileges by default. The right to drop an object, or to alter its definition in any way is not described by a grantable privilege; it is inherent in the owner, and cannot be granted or revoked. The owner implicitly has all grant options for the object, too.
+
+Depending on the type of object, the initial default privileges may include granting some privileges to `PUBLIC`. The default is no public access for tables, schemas, and tablespaces; `CONNECT` privilege and `TEMP` table creation privilege for databases; `EXECUTE` privilege for functions; and `USAGE` privilege for languages. The object owner may of course revoke these privileges.
+
+**GRANT on Roles**
+
+This variant of the `GRANT` command grants membership in a role to one or more other roles. Membership in a role is significant because it conveys the privileges granted to a role to each of its members.
+
+If `WITH ADMIN OPTION` is specified, the member may in turn grant membership in the role to others, and revoke membership in the role as well. Database superusers can grant or revoke membership in any role to anyone. Roles having `CREATEROLE` privilege can grant or revoke membership in any role that is not a superuser.
+
+Unlike the case with privileges, membership in a role cannot be granted to `PUBLIC`.
+
+## Parameters<a id="topic1__section7"></a>
+
+SELECT
+Allows `SELECT` from any column of the specified table, view, or sequence. Also allows the use of `COPY TO`. For sequences, this privilege also allows the use of the `currval` function.
+
+INSERT
+Allows `INSERT` of a new row into the specified table. Also allows `COPY FROM`.
+
+UPDATE
+Allows `UPDATE` of any column of the specified table. `SELECT ... FOR UPDATE` and `SELECT ... FOR SHARE` also require this privilege (as well as the `SELECT` privilege). For sequences, this privilege allows the use of the `nextval` and `setval` functions.
+
+DELETE
+Allows `DELETE` of a row from the specified table.
+
+REFERENCES
+This keyword is accepted, although foreign key constraints are currently not supported in HAWQ. To create a foreign key constraint, it is necessary to have this privilege on both the referencing and referenced tables.
+
+TRIGGER
+Allows the creation of a trigger on the specified table.
+
+**Note:** HAWQ does not support triggers.
+
+CREATE
+For databases, allows new schemas to be created within the database.
+
+For schemas, allows new objects to be created within the schema. To rename an existing object, you must own the object and have this privilege for the containing schema.
+
+For tablespaces, allows tables and indexes to be created within the tablespace, and allows databases to be created that have the tablespace as their default tablespace. (Note that revoking this privilege will not alter the placement of existing objects.)
+
+CONNECT
+Allows the user to connect to the specified database. This privilege is checked at connection startup (in addition to checking any restrictions imposed by `pg_hba.conf`).
+
+TEMPORARY
+TEMP
+Allows temporary tables to be created while using the database.
+
+EXECUTE
+Allows the use of the specified function and the use of any operators that are implemented on top of the function. This is the only type of privilege that is applicable to functions. (This syntax works for aggregate functions, as well.)
+
+USAGE
+For procedural languages, allows the use of the specified language for the creation of functions in that language. This is the only type of privilege that is applicable to procedural languages.
+
+For schemas, allows access to objects contained in the specified schema (assuming that the objects' own privilege requirements are also met). Essentially this allows the grantee to look up objects within the schema.
+
+For sequences, this privilege allows the use of the `currval` and `nextval` functions.
+
+ALL PRIVILEGES
+Grant all of the available privileges at once. The `PRIVILEGES` key word is optional in HAWQ, though it is required by strict SQL.
+
+PUBLIC
+A special group-level role that denotes that the privileges are to be granted to all roles, including those that may be created later.
+
+WITH GRANT OPTION
+The recipient of the privilege may in turn grant it to others.
+
+WITH ADMIN OPTION
+The member of a role may in turn grant membership in the role to others.
+
+## Notes<a id="topic1__section8"></a>
+
+Database superusers can access all objects regardless of object privilege settings. One exception to this rule is view objects. Access to tables referenced in the view is determined by permissions of the view owner not the current user (even if the current user is a superuser).
+
+If a superuser chooses to issue a `GRANT` or `REVOKE` command, the command is performed as though it were issued by the owner of the affected object. In particular, privileges granted via such a command will appear to have been granted by the object owner. For role membership, the membership appears to have been granted by the containing role itself.
+
+`GRANT` and `REVOKE` can also be done by a role that is not the owner of the affected object, but is a member of the role that owns the object, or is a member of a role that holds privileges `WITH GRANT OPTION` on the object. In this case the privileges will be recorded as having been granted by the role that actually owns the object or holds the privileges `WITH GRANT OPTION`.
+
+Granting permission on a table does not automatically extend permissions to any sequences used by the table, including sequences tied to `SERIAL` columns. Permissions on a sequence must be set separately.
+
+HAWQ does not support granting or revoking privileges for individual columns of a table. One possible workaround is to create a view having just the desired columns and then grant privileges to that view.
+
+Use psql's `\z` meta-command to obtain information about existing privileges for an object.
+
+## Examples<a id="topic1__section9"></a>
+
+Grant insert privilege to all roles on table `mytable`:
+
+``` pre
+GRANT INSERT ON mytable TO PUBLIC;
+```
+
+Grant all available privileges to role `sally` on the view `topten`. Note that while the above will indeed grant all privileges if executed by a superuser or the owner of `topten`, when executed by someone else it will only grant those permissions for which the granting role has grant options.
+
+``` pre
+GRANT ALL PRIVILEGES ON topten TO sally;
+```
+
+Grant membership in role `admins` to user `joe`:
+
+``` pre
+GRANT admins TO joe;
+```
+
+## Compatibility<a id="topic1__section10"></a>
+
+The `PRIVILEGES` key word in is required in the SQL standard, but optional in HAWQ. The SQL standard does not support setting the privileges on more than one object per command.
+
+HAWQ allows an object owner to revoke his own ordinary privileges: for example, a table owner can make the table read-only to himself by revoking his own `INSERT`, `UPDATE`, and `DELETE` privileges. This is not possible according to the SQL standard. HAWQ treats the owner's privileges as having been granted by the owner to himself; therefore he can revoke them too. In the SQL standard, the owner's privileges are granted by an assumed *system* entity.
+
+The SQL standard allows setting privileges for individual columns within a table.
+
+The SQL standard provides for a `USAGE` privilege on other kinds of objects: character sets, collations, translations, domains.
+
+Privileges on databases, tablespaces, schemas, and languages are HAWQ extensions.
+
+## See Also<a id="topic1__section11"></a>
+
+[REVOKE](REVOKE.html)
diff --git a/reference/sql/INSERT.html.md.erb b/reference/sql/INSERT.html.md.erb
new file mode 100644
index 0000000..a2d753a
--- /dev/null
+++ b/reference/sql/INSERT.html.md.erb
@@ -0,0 +1,112 @@
+---
+title: INSERT
+---
+
+Creates new rows in a table.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+INSERT INTO table [( column [, ...] )]
+ {DEFAULT VALUES | VALUES ( {expression | DEFAULT} [, ...] )
+ [, ...] | query}
+```
+
+## Description<a id="topic1__section3"></a>
+
+`INSERT` inserts new rows into a table. One can insert one or more rows specified by value expressions, or zero or more rows resulting from a query.
+
+The target column names may be listed in any order. If no list of column names is given at all, the default is the columns of the table in their declared order. The values supplied by the `VALUES` clause or query are associated with the explicit or implicit column list left-to-right.
+
+Each column not present in the explicit or implicit column list will be filled with a default value, either its declared default value or null if there is no default.
+
+If the expression for any column is not of the correct data type, automatic type conversion will be attempted.
+
+You must have `INSERT` privilege on a table in order to insert into it.
+
+**Note:** HAWQ supports 127 concurrent inserts currently.
+
+**Outputs**
+On successful completion, an `INSERT` command returns a command tag of the form:
+
+``` pre
+INSERT oid
+ count
+
+```
+
+The *count* is the number of rows inserted. If count is exactly one, and the target table has OIDs, then *oid* is the OID assigned to the inserted row. Otherwise *oid* is zero.
+
+## Parameters<a id="topic1__section5"></a>
+
+ *table*
+The name (optionally schema-qualified) of an existing table.
+
+ *column*
+The name of a column in table. The column name can be qualified with a subfield name or array subscript, if needed. (Inserting into only some fields of a composite column leaves the other fields null.)
+
+DEFAULT VALUES
+All columns will be filled with their default values.
+
+ *expression*
+An expression or value to assign to the corresponding column.
+
+DEFAULT
+The corresponding column will be filled with its default value.
+
+ *query*
+A query (`SELECT` statement) that supplies the rows to be inserted. Refer to the [SELECT](SELECT.html) statement for a description of the syntax.
+
+## Examples<a id="topic1__section7"></a>
+
+Insert a single row into table `films`:
+
+``` pre
+INSERT INTO films VALUES ('UA502', 'Bananas', 105,
+'1971-07-13', 'Comedy', '82 minutes');
+```
+
+In this example, the `length` column is omitted and therefore it will have the default value:
+
+``` pre
+INSERT INTO films (code, title, did, date_prod, kind) VALUES
+('T_601', 'Yojimbo', 106, '1961-06-16', 'Drama');
+```
+
+This example uses the `DEFAULT` clause for the `date_prod` column rather than specifying a value:
+
+``` pre
+INSERT INTO films VALUES ('UA502', 'Bananas', 105, DEFAULT,
+'Comedy', '82 minutes');
+```
+
+To insert a row consisting entirely of default values:
+
+``` pre
+INSERT INTO films DEFAULT VALUES;
+```
+
+To insert multiple rows using the multirow `VALUES` syntax:
+
+``` pre
+INSERT INTO films (code, title, did, date_prod, kind) VALUES
+ ('B6717', 'Tampopo', 110, '1985-02-10', 'Comedy'),
+ ('HG120', 'The Dinner Game', 140, DEFAULT, 'Comedy');
+```
+
+This example inserts some rows into table `films` from a table `tmp_films` with the same column layout as `films`:
+
+``` pre
+INSERT INTO films SELECT * FROM tmp_films WHERE date_prod <
+'2004-05-07';
+```
+
+## Compatibility<a id="topic1__section8"></a>
+
+`INSERT` conforms to the SQL standard. The case in which a column name list is omitted, but not all the columns are filled from the `VALUES` clause or query, is disallowed by the standard.
+
+Possible limitations of the *query* clause are documented under `SELECT`.
+
+## See Also<a id="topic1__section9"></a>
+
+[COPY](COPY.html), [SELECT](SELECT.html), [CREATE EXTERNAL TABLE](CREATE-EXTERNAL-TABLE.html)
diff --git a/reference/sql/PREPARE.html.md.erb b/reference/sql/PREPARE.html.md.erb
new file mode 100644
index 0000000..cab1259
--- /dev/null
+++ b/reference/sql/PREPARE.html.md.erb
@@ -0,0 +1,67 @@
+---
+title: PREPARE
+---
+
+Prepare a statement for execution.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+PREPARE name [ (datatype [, ...] ) ] AS statement
+
+```
+
+## Description<a id="topic1__section3"></a>
+
+`PREPARE` creates a prepared statement, possibly with unbound parameters. A prepared statement is a server-side object that can be used to optimize performance. A prepared statement may be subsequently executed with a binding for its parameters. HAWQ may choose to replan the query for different executions of the same prepared statement.
+
+Prepared statements can take parameters: values that are substituted into the statement when it is executed. When creating the prepared statement, refer to parameters by position, using `$1`, `$2`, etc. A corresponding list of parameter data types can optionally be specified. When a parameter's data type is not specified or is declared as unknown, the type is inferred from the context in which the parameter is used (if possible). When executing the statement, specify the actual values for these parameters in the `EXECUTE` statement.
+
+Prepared statements only last for the duration of the current database session. When the session ends, the prepared statement is forgotten, so it must be recreated before being used again. This also means that a single prepared statement cannot be used by multiple simultaneous database clients; however, each client can create their own prepared statement to use. The prepared statement can be manually cleaned up using the [DEALLOCATE](DEALLOCATE.html) command.
+
+Prepared statements have the largest performance advantage when a single session is being used to execute a large number of similar statements. The performance difference will be particularly significant if the statements are complex to plan or rewrite, for example, if the query involves a join of many tables or requires the application of several rules. If the statement is relatively simple to plan and rewrite but relatively expensive to execute, the performance advantage of prepared statements will be less noticeable.
+
+## Parameters<a id="topic1__section4"></a>
+
+ *name*
+An arbitrary name given to this particular prepared statement. It must be unique within a single session and is subsequently used to execute or deallocate a previously prepared statement.
+
+ *datatype*
+The data type of a parameter to the prepared statement. If the data type of a particular parameter is unspecified or is specified as unknown, it will be inferred from the context in which the parameter is used. To refer to the parameters in the prepared statement itself, use `$1`, `$2`, etc.
+
+ *statement*
+Any `SELECT`, `INSERT`, `UPDATE`, `DELETE`, or `VALUES` statement.
+
+## Notes<a id="topic1__section5"></a>
+
+In some situations, the query plan produced for a prepared statement will be inferior to the query plan that would have been chosen if the statement had been submitted and executed normally. This is because when the statement is planned and the planner attempts to determine the optimal query plan, the actual values of any parameters specified in the statement are unavailable. HAWQ collects statistics on the distribution of data in the table, and can use constant values in a statement to make guesses about the likely result of executing the statement. Since this data is unavailable when planning prepared statements with parameters, the chosen plan may be suboptimal. To examine the query plan HAWQ has chosen for a prepared statement, use `EXPLAIN`.
+
+For more information on query planning and the statistics collected by HAWQ for that purpose, see the `ANALYZE` documentation.
+
+You can see all available prepared statements of a session by querying the `pg_prepared_statements` system view.
+
+## Examples<a id="topic1__section6"></a>
+
+Create a prepared statement for an `INSERT` statement, and then execute it:
+
+``` pre
+PREPARE fooplan (int, text, bool, numeric) AS INSERT INTO
+foo VALUES($1, $2, $3, $4);
+EXECUTE fooplan(1, 'Hunter Valley', 't', 200.00);
+```
+
+Create a prepared statement for a `SELECT` statement, and then execute it. Note that the data type of the second parameter is not specified, so it is inferred from the context in which `$2` is used:
+
+``` pre
+PREPARE usrrptplan (int) AS SELECT * FROM users u, logs l
+WHERE u.usrid=$1 AND u.usrid=l.usrid AND l.date = $2;
+EXECUTE usrrptplan(1, current_date);
+```
+
+## Compatibility<a id="topic1__section7"></a>
+
+The SQL standard includes a `PREPARE` statement, but it is only for use in embedded SQL. This version of the `PREPARE` statement also uses a somewhat different syntax.
+
+## See Also<a id="topic1__section8"></a>
+
+[EXECUTE](EXECUTE.html), [DEALLOCATE](DEALLOCATE.html)
diff --git a/reference/sql/REASSIGN-OWNED.html.md.erb b/reference/sql/REASSIGN-OWNED.html.md.erb
new file mode 100644
index 0000000..c98c37b
--- /dev/null
+++ b/reference/sql/REASSIGN-OWNED.html.md.erb
@@ -0,0 +1,48 @@
+---
+title: REASSIGN OWNED
+---
+
+Changes the ownership of database objects owned by a database role.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+REASSIGN OWNED BY old_role [, ...] TO new_role
+
+```
+
+## Description<a id="topic1__section3"></a>
+
+`REASSIGN OWNED` reassigns all the objects in the current database that are owned by *old\_row* to *new\_role*. Note that it does not change the ownership of the database itself.
+
+## Parameters<a id="topic1__section4"></a>
+
+ *old\_role*
+The name of a role. The ownership of all the objects in the current database owned by this role will be reassigned to *new\_role*.
+
+ *new\_role*
+The name of the role that will be made the new owner of the affected objects.
+
+## Notes<a id="topic1__section5"></a>
+
+`REASSIGN OWNED` is often used to prepare for the removal of one or more roles. Because `REASSIGN OWNED` only affects the objects in the current database, it is usually necessary to execute this command in each database that contains objects owned by a role that is to be removed.
+
+The `DROP OWNED` command is an alternative that drops all the database objects owned by one or more roles.
+
+The `REASSIGN OWNED` command does not affect the privileges granted to the old roles in objects that are not owned by them. Use `DROP OWNED` to revoke those privileges.
+
+## Examples<a id="topic1__section6"></a>
+
+Reassign any database objects owned by the role named `sally` and `bob` to `admin`;
+
+``` pre
+REASSIGN OWNED BY sally, bob TO admin;
+```
+
+## Compatibility<a id="topic1__section7"></a>
+
+The `REASSIGN OWNED` statement is a HAWQ extension.
+
+## See Also<a id="topic1__section8"></a>
+
+[DROP OWNED](DROP-OWNED.html), [DROP ROLE](DROP-ROLE.html)
diff --git a/reference/sql/RELEASE-SAVEPOINT.html.md.erb b/reference/sql/RELEASE-SAVEPOINT.html.md.erb
new file mode 100644
index 0000000..0603615
--- /dev/null
+++ b/reference/sql/RELEASE-SAVEPOINT.html.md.erb
@@ -0,0 +1,48 @@
+---
+title: RELEASE SAVEPOINT
+---
+
+Destroys a previously defined savepoint.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+RELEASE [SAVEPOINT] savepoint_name
+
+```
+
+## Description<a id="topic1__section3"></a>
+
+`RELEASE SAVEPOINT` destroys a savepoint previously defined in the current transaction.
+
+Destroying a savepoint makes it unavailable as a rollback point, but it has no other user visible behavior. It does not undo the effects of commands executed after the savepoint was established. (To do that, see [ROLLBACK TO SAVEPOINT](ROLLBACK-TO-SAVEPOINT.html).) Destroying a savepoint when it is no longer needed may allow the system to reclaim some resources earlier than transaction end.
+
+`RELEASE SAVEPOINT` also destroys all savepoints that were established *after* the named savepoint was established.
+
+## Parameters<a id="topic1__section4"></a>
+
+ *savepoint\_name*
+The name of the savepoint to destroy.
+
+## Examples<a id="topic1__section5"></a>
+
+To establish and later destroy a savepoint:
+
+``` pre
+BEGIN;
+ INSERT INTO table1 VALUES (3);
+ SAVEPOINT my_savepoint;
+ INSERT INTO table1 VALUES (4);
+ RELEASE SAVEPOINT my_savepoint;
+COMMIT;
+```
+
+The above transaction will insert both 3 and 4.
+
+## Compatibility<a id="topic1__section6"></a>
+
+This command conforms to the SQL standard. The standard specifies that the key word `SAVEPOINT` is mandatory, but HAWQ allows it to be omitted.
+
+## See Also<a id="topic1__section7"></a>
+
+[BEGIN](BEGIN.html), [SAVEPOINT](SAVEPOINT.html), [ROLLBACK TO SAVEPOINT](ROLLBACK-TO-SAVEPOINT.html), [COMMIT](COMMIT.html)
diff --git a/reference/sql/RESET.html.md.erb b/reference/sql/RESET.html.md.erb
new file mode 100644
index 0000000..8f16fbd
--- /dev/null
+++ b/reference/sql/RESET.html.md.erb
@@ -0,0 +1,45 @@
+---
+title: RESET
+---
+
+Restores the value of a system configuration parameter to the default value.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+RESET configuration_parameter
+
+RESET ALL
+```
+
+## Description<a id="topic1__section3"></a>
+
+`RESET` restores system configuration parameters to their default values. `RESET` is an alternative spelling for `SET configuration_parameter TO DEFAULT`.
+
+The default value is defined as the value that the parameter would have had, had no `SET` ever been issued for it in the current session. The actual source of this value might be a compiled-in default, the master `hawq-site.xml` configuration file, command-line options, or per-database or per-user default settings.
+
+See [Server Configuration Parameter Reference](../HAWQSiteConfig.html) for more information.
+
+## Parameters<a id="topic1__section4"></a>
+
+ *configuration\_parameter*
+The name of a system configuration parameter. See [Server Configuration Parameter Reference](../HAWQSiteConfig.html) for a list of configuration parameters.
+
+ALL
+Resets all settable configuration parameters to their default values.
+
+## Examples<a id="topic1__section5"></a>
+
+Set the `default_statement_mem` configuration parameter to its default value:
+
+``` pre
+RESET default_statement_mem;
+```
+
+## Compatibility<a id="topic1__section6"></a>
+
+`RESET` is a HAWQ extension.
+
+## See Also<a id="topic1__section7"></a>
+
+[SET](SET.html)
diff --git a/reference/sql/REVOKE.html.md.erb b/reference/sql/REVOKE.html.md.erb
new file mode 100644
index 0000000..daef02b
--- /dev/null
+++ b/reference/sql/REVOKE.html.md.erb
@@ -0,0 +1,101 @@
+---
+title: REVOKE
+---
+
+Removes access privileges.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+REVOKE [GRANT OPTION FOR] { {SELECT | INSERT | UPDATE | DELETE
+ | REFERENCES | TRUNCATE } [,...] | ALL [PRIVILEGES] }
+ ON [TABLE] tablename [, ...]
+ FROM {rolename | PUBLIC} [, ...]
+ [CASCADE | RESTRICT]
+
+REVOKE [GRANT OPTION FOR] { {USAGE | SELECT | UPDATE} [,...]
+ | ALL [PRIVILEGES] }
+ ON SEQUENCE sequencename [, ...]
+ FROM { rolename | PUBLIC } [, ...]
+ [CASCADE | RESTRICT]
+
+REVOKE [GRANT OPTION FOR] { {CREATE | CONNECT
+ | TEMPORARY | TEMP} [,...] | ALL [PRIVILEGES] }
+ ON DATABASE dbname [, ...]
+ FROM {rolename | PUBLIC} [, ...]
+ [CASCADE | RESTRICT]
+
+REVOKE [GRANT OPTION FOR] {EXECUTE | ALL [PRIVILEGES]}
+ ON FUNCTION funcname ( [[argmode] [argname] argtype
+ [, ...]] ) [, ...]
+ FROM {rolename | PUBLIC} [, ...]
+ [CASCADE | RESTRICT]
+
+REVOKE [GRANT OPTION FOR] {USAGE | ALL [PRIVILEGES]}
+ ON LANGUAGE langname [, ...]
+ FROM {rolename | PUBLIC} [, ...]
+ [ CASCADE | RESTRICT ]
+
+REVOKE [GRANT OPTION FOR] { {CREATE | USAGE} [,...]
+ | ALL [PRIVILEGES] }
+ ON SCHEMA schemaname [, ...]
+ FROM {rolename | PUBLIC} [, ...]
+ [CASCADE | RESTRICT]
+
+REVOKE [GRANT OPTION FOR] { CREATE | ALL [PRIVILEGES] }
+ ON TABLESPACE tablespacename [, ...]
+ FROM { rolename | PUBLIC } [, ...]
+ [CASCADE | RESTRICT]
+
+REVOKE [ADMIN OPTION FOR] parent_role [, ...]
+ FROM member_role [, ...]
+ [CASCADE | RESTRICT]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`REVOKE` command revokes previously granted privileges from one or more roles. The key word `PUBLIC` refers to the implicitly defined group of all roles.
+
+See the description of the [GRANT](GRANT.html) command for the meaning of the privilege types.
+
+Note that any particular role will have the sum of privileges granted directly to it, privileges granted to any role it is presently a member of, and privileges granted to `PUBLIC`. Thus, for example, revoking `SELECT` privilege from `PUBLIC` does not necessarily mean that all roles have lost `SELECT` privilege on the object: those who have it granted directly or via another role will still have it.
+
+If `GRANT OPTION FOR` is specified, only the grant option for the privilege is revoked, not the privilege itself. Otherwise, both the privilege and the grant option are revoked.
+
+If a role holds a privilege with grant option and has granted it to other roles then the privileges held by those other roles are called dependent privileges. If the privilege or the grant option held by the first role is being revoked and dependent privileges exist, those dependent privileges are also revoked if `CASCADE` is specified, else the revoke action will fail. This recursive revocation only affects privileges that were granted through a chain of roles that is traceable to the role that is the subject of this `REVOKE` command. Thus, the affected roles may effectively keep the privilege if it was also granted through other roles.
+
+When revoking membership in a role, `GRANT OPTION` is instead called `ADMIN OPTION`, but the behavior is similar.
+
+## Parameters<a id="topic1__section4"></a>
+
+See [GRANT](GRANT.html).
+
+## Examples<a id="topic1__section5"></a>
+
+Revoke insert privilege for the public on table `films`:
+
+``` pre
+REVOKE INSERT ON films FROM PUBLIC;
+```
+
+Revoke all privileges from role `sally` on view `topten`. Note that this actually means revoke all privileges that the current role granted (if not a superuser).
+
+``` pre
+REVOKE ALL PRIVILEGES ON topten FROM sally;
+```
+
+Revoke membership in role `admins` from user `joe`:
+
+``` pre
+REVOKE admins FROM joe;
+```
+
+## Compatibility<a id="topic1__section6"></a>
+
+The compatibility notes of the [GRANT](GRANT.html) command also apply to `REVOKE`.
+
+Either `RESTRICT` or `CASCADE` is required according to the standard, but HAWQ assumes `RESTRICT` by default.
+
+## See Also<a id="topic1__section7"></a>
+
+[GRANT](GRANT.html)
diff --git a/reference/sql/ROLLBACK-TO-SAVEPOINT.html.md.erb b/reference/sql/ROLLBACK-TO-SAVEPOINT.html.md.erb
new file mode 100644
index 0000000..5110f5d
--- /dev/null
+++ b/reference/sql/ROLLBACK-TO-SAVEPOINT.html.md.erb
@@ -0,0 +1,69 @@
+---
+title: ROLLBACK TO SAVEPOINT
+---
+
+Rolls back the current transaction to a savepoint.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+ROLLBACK [WORK | TRANSACTION] TO [SAVEPOINT] savepoint_name
+
+```
+
+## Description<a id="topic1__section3"></a>
+
+This command will roll back all commands that were executed after the savepoint was established. The savepoint remains valid and can be rolled back to again later, if needed.
+
+`ROLLBACK TO SAVEPOINT` implicitly destroys all savepoints that were established after the named savepoint.
+
+## Parameters<a id="topic1__section4"></a>
+
+WORK
+TRANSACTION
+Optional key words. They have no effect.
+
+ *savepoint\_name*
+The name of a savepoint to roll back to.
+
+## Notes<a id="topic1__section5"></a>
+
+Use `RELEASE SAVEPOINT` to destroy a savepoint without discarding the effects of commands executed after it was established.
+
+Specifying a savepoint name that has not been established is an error.
+
+Cursors have somewhat non-transactional behavior with respect to savepoints. Any cursor that is opened inside a savepoint will be closed when the savepoint is rolled back. If a previously opened cursor is affected by a `FETCH` command inside a savepoint that is later rolled back, the cursor position remains at the position that `FETCH` left it pointing to (that is, `FETCH` is not rolled back). Closing a cursor is not undone by rolling back, either. A cursor whose execution causes a transaction to abort is put in a can't-execute state, so while the transaction can be restored using `ROLLBACK TO SAVEPOINT`, the cursor can no longer be used.
+
+## Examples<a id="topic1__section6"></a>
+
+To undo the effects of the commands executed after `my_savepoint` was established:
+
+``` pre
+ROLLBACK TO SAVEPOINT my_savepoint;
+```
+
+Cursor positions are not affected by a savepoint rollback:
+
+``` pre
+BEGIN;
+DECLARE foo CURSOR FOR SELECT 1 UNION SELECT 2;
+SAVEPOINT foo;
+FETCH 1 FROM foo;
+column
+----------
+ 1
+ROLLBACK TO SAVEPOINT foo;
+FETCH 1 FROM foo;
+column
+----------
+ 2
+COMMIT;
+```
+
+## Compatibility<a id="topic1__section7"></a>
+
+The SQL standard specifies that the key word `SAVEPOINT` is mandatory, but HAWQ (and Oracle) allow it to be omitted. SQL allows only `WORK`, not `TRANSACTION`, as a stopword after `ROLLBACK`. Also, SQL has an optional clause `AND [NO] CHAIN` which is not currently supported by HAWQ. Otherwise, this command conforms to the SQL standard.
+
+## See Also<a id="topic1__section8"></a>
+
+[BEGIN](BEGIN.html), [COMMIT](COMMIT.html), [SAVEPOINT](SAVEPOINT.html), [RELEASE SAVEPOINT](RELEASE-SAVEPOINT.html), [ROLLBACK](ROLLBACK.html)
diff --git a/reference/sql/ROLLBACK.html.md.erb b/reference/sql/ROLLBACK.html.md.erb
new file mode 100644
index 0000000..80defc9
--- /dev/null
+++ b/reference/sql/ROLLBACK.html.md.erb
@@ -0,0 +1,43 @@
+---
+title: ROLLBACK
+---
+
+Aborts the current transaction.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+ROLLBACK [WORK | TRANSACTION]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`ROLLBACK` rolls back the current transaction and causes all the updates made by the transaction to be discarded.
+
+## Parameters<a id="topic1__section4"></a>
+
+WORK
+TRANSACTION
+Optional key words. They have no effect.
+
+## Notes<a id="topic1__section5"></a>
+
+Use `COMMIT` to successfully end the current transaction.
+
+Issuing `ROLLBACK` when not inside a transaction does no harm, but it will provoke a warning message.
+
+## Examples<a id="topic1__section6"></a>
+
+To discard all changes made in the current transaction:
+
+``` pre
+ROLLBACK;
+```
+
+## Compatibility<a id="topic1__section7"></a>
+
+The SQL standard only specifies the two forms `ROLLBACK` and `ROLLBACK WORK`. Otherwise, this command is fully conforming.
+
+## See Also<a id="topic1__section8"></a>
+
+[BEGIN](BEGIN.html), [COMMIT](COMMIT.html), [SAVEPOINT](SAVEPOINT.html), [ROLLBACK TO SAVEPOINT](ROLLBACK-TO-SAVEPOINT.html)
diff --git a/reference/sql/SAVEPOINT.html.md.erb b/reference/sql/SAVEPOINT.html.md.erb
new file mode 100644
index 0000000..8ed61f3
--- /dev/null
+++ b/reference/sql/SAVEPOINT.html.md.erb
@@ -0,0 +1,66 @@
+---
+title: SAVEPOINT
+---
+
+Defines a new savepoint within the current transaction.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+SAVEPOINT savepoint_name
+
+```
+
+## Description<a id="topic1__section3"></a>
+
+`SAVEPOINT` establishes a new savepoint within the current transaction.
+
+A savepoint is a special mark inside a transaction that allows all commands that are executed after it was established to be rolled back, restoring the transaction state to what it was at the time of the savepoint.
+
+## Parameters<a id="topic1__section4"></a>
+
+ *savepoint\_name*
+The name of the new savepoint.
+
+## Notes<a id="topic1__section5"></a>
+
+Use [ROLLBACK TO SAVEPOINT](ROLLBACK-TO-SAVEPOINT.html) to rollback to a savepoint. Use [RELEASE SAVEPOINT](RELEASE-SAVEPOINT.html) to destroy a savepoint, keeping the effects of commands executed after it was established.
+
+Savepoints can only be established when inside a transaction block. There can be multiple savepoints defined within a transaction.
+
+## Examples<a id="topic1__section6"></a>
+
+To establish a savepoint and later undo the effects of all commands executed after it was established:
+
+``` pre
+BEGIN;
+ INSERT INTO table1 VALUES (1);
+ SAVEPOINT my_savepoint;
+ INSERT INTO table1 VALUES (2);
+ ROLLBACK TO SAVEPOINT my_savepoint;
+ INSERT INTO table1 VALUES (3);
+COMMIT;
+```
+
+The above transaction will insert the values 1 and 3, but not 2.
+
+To establish and later destroy a savepoint:
+
+``` pre
+BEGIN;
+ INSERT INTO table1 VALUES (3);
+ SAVEPOINT my_savepoint;
+ INSERT INTO table1 VALUES (4);
+ RELEASE SAVEPOINT my_savepoint;
+COMMIT;
+```
+
+The above transaction will insert both 3 and 4.
+
+## Compatibility<a id="topic1__section7"></a>
+
+SQL requires a savepoint to be destroyed automatically when another savepoint with the same name is established. In HAWQ, the old savepoint is kept, though only the more recent one will be used when rolling back or releasing. (Releasing the newer savepoint will cause the older one to again become accessible to [ROLLBACK TO SAVEPOINT](ROLLBACK-TO-SAVEPOINT.html) and [RELEASE SAVEPOINT](RELEASE-SAVEPOINT.html).) Otherwise, `SAVEPOINT` is fully SQL conforming.
+
+## See Also<a id="topic1__section8"></a>
+
+[BEGIN](BEGIN.html), [COMMIT](COMMIT.html), [ROLLBACK](ROLLBACK.html), [RELEASE SAVEPOINT](RELEASE-SAVEPOINT.html), [ROLLBACK TO SAVEPOINT](ROLLBACK-TO-SAVEPOINT.html)
diff --git a/reference/sql/SELECT-INTO.html.md.erb b/reference/sql/SELECT-INTO.html.md.erb
new file mode 100644
index 0000000..3d1074d
--- /dev/null
+++ b/reference/sql/SELECT-INTO.html.md.erb
@@ -0,0 +1,55 @@
+---
+title: SELECT INTO
+---
+
+Defines a new table from the results of a query.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+SELECT [ALL | DISTINCT [ON ( expression [, ...] )]]
+ * | expression [AS output_name] [, ...]
+ INTO [TEMPORARY | TEMP] [TABLE] new_table
+ [FROM from_item [, ...]]
+ [WHERE condition]
+ [GROUP BY expression [, ...]]
+ [HAVING condition [, ...]]
+ [{UNION | INTERSECT | EXCEPT} [ALL] select]
+ [ORDER BY expression [ASC | DESC | USING operator] [, ...]]
+ [LIMIT {count | ALL}]
+ [OFFSET start]
+ [FOR {UPDATE | SHARE} [OF table_name [, ...]] [NOWAIT]
+ [...]]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`SELECT INTO` creates a new table and fills it with data computed by a query. The data is not returned to the client, as it is with a normal `SELECT`. The new table's columns have the names and data types associated with the output columns of the `SELECT`. Data is always distributed randomly.
+
+## Parameters<a id="topic1__section4"></a>
+
+The majority of parameters for `SELECT INTO` are the same as [SELECT](SELECT.html).
+
+TEMPORARY,
+TEMP
+If specified, the table is created as a temporary table.
+
+ *new\_table*
+The name (optionally schema-qualified) of the table to be created.
+
+## Examples<a id="topic1__section5"></a>
+
+Create a new table `films_recent` consisting of only recent entries from the table `films`:
+
+``` pre
+SELECT * INTO films_recent FROM films WHERE date_prod >=
+'2006-01-01';
+```
+
+## Compatibility<a id="topic1__section6"></a>
+
+The SQL standard uses `SELECT INTO` to represent selecting values into scalar variables of a host program, rather than creating a new table. The HAWQ usage of `SELECT INTO` to represent table creation is historical. It is best to use [CREATE TABLE AS](CREATE-TABLE-AS.html) for this purpose in new applications.
+
+## See Also<a id="topic1__section7"></a>
+
+[SELECT](SELECT.html), [CREATE TABLE AS](CREATE-TABLE-AS.html)
diff --git a/reference/sql/SELECT.html.md.erb b/reference/sql/SELECT.html.md.erb
new file mode 100644
index 0000000..11d823d
--- /dev/null
+++ b/reference/sql/SELECT.html.md.erb
@@ -0,0 +1,512 @@
+---
+title: SELECT
+---
+
+Retrieves rows from a table or view.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+SELECT [ALL | DISTINCT [ON (expression [, ...])]]
+ * | expression [[AS] output_name] [, ...]
+ [FROM from_item [, ...]]
+ [WHERE condition]
+ [GROUP BY grouping_element [, ...]]
+ [HAVING condition [, ...]]
+ [WINDOW window_name AS (window_specification)]
+ [{UNION | INTERSECT | EXCEPT} [ALL] select]
+ [ORDER BY expression [ASC | DESC | USING operator] [, ...]]
+ [LIMIT {count | ALL}]
+ [OFFSET start]
+```
+
+where *grouping\_element* can be one of:
+
+``` pre
+ ()
+ expression
+ ROLLUP (expression [,...])
+ CUBE (expression [,...])
+ GROUPING SETS ((grouping_element [, ...]))
+```
+
+where *window\_specification* can be:
+
+``` pre
+ [window_name]
+ [PARTITION BY expression [, ...]]
+ [ORDER BY expression [ASC | DESC | USING operator] [, ...]
+ [{RANGE | ROWS}
+ { UNBOUNDED PRECEDING
+ | expression PRECEDING
+ | CURRENT ROW
+ | BETWEEN window_frame_bound AND window_frame_bound }]]
+ where window_frame_bound can be one of:
+ UNBOUNDED PRECEDING
+ expression PRECEDING
+ CURRENT ROW
+ expression FOLLOWING
+ UNBOUNDED FOLLOWING
+```
+
+where *from\_item* can be one of:
+
+``` pre
+[ONLY] table_name [[AS] alias [( column_alias [, ...] )]]
+(select) [AS] alias [( column_alias [, ...] )]
+function_name ( [argument [, ...]] ) [AS] alias
+ [( column_alias [, ...]
+ | column_definition [, ...] )]
+function_name ( [argument [, ...]] ) AS
+ ( column_definition [, ...] )
+from_item [NATURAL] join_type
+ from_item
+ [ON join_condition | USING ( join_column [, ...] )]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`SELECT` retrieves rows from zero or more tables. The general processing of `SELECT` is as follows:
+
+1. All elements in the `FROM` list are computed. (Each element in the `FROM` list is a real or virtual table.) If more than one element is specified in the `FROM` list, they are cross-joined together.
+2. If the `WHERE` clause is specified, all rows that do not satisfy the condition are eliminated from the output.
+3. If the `GROUP BY` clause is specified, the output is divided into groups of rows that match on one or more of the defined grouping elements. If the `HAVING` clause is present, it eliminates groups that do not satisfy the given condition.
+4. If a window expression is specified (and optional `WINDOW` clause), the output is organized according to the positional (row) or value-based (range) window frame.
+5. `DISTINCT` eliminates duplicate rows from the result. `DISTINCT ON` eliminates rows that match on all the specified expressions. `ALL` (the default) will return all candidate rows, including duplicates.
+6. The actual output rows are computed using the `SELECT` output expressions for each selected row.
+7. Using the operators `UNION`, `INTERSECT`, and `EXCEPT`, the output of more than one `SELECT` statement can be combined to form a single result set. The `UNION` operator returns all rows that are in one or both of the result sets. The `INTERSECT` operator returns all rows that are strictly in both result sets. The `EXCEPT` operator returns the rows that are in the first result set but not in the second. In all three cases, duplicate rows are eliminated unless `ALL` is specified.
+8. If the `ORDER BY` clause is specified, the returned rows are sorted in the specified order. If `ORDER BY` is not given, the rows are returned in whatever order the system finds fastest to produce.
+9. If the `LIMIT` or `OFFSET` clause is specified, the `SELECT` statement only returns a subset of the result rows.
+
+You must have `SELECT` privilege on a table to read its values.
+
+## Parameters<a id="topic1__section4"></a>
+
+**The SELECT List**
+
+The `SELECT` list (between the key words `SELECT` and `FROM`) specifies expressions that form the output rows of the `SELECT` statement. The expressions can (and usually do) refer to columns computed in the `FROM` clause.
+
+Using the clause `[AS] ` *output\_name*, another name can be specified for an output column. This name is primarily used to label the column for display. It can also be used to refer to the column's value in `ORDER BY` and `GROUP BY` clauses, but not in the `WHERE` or `HAVING` clauses; there you must write out the expression instead. The `AS` keyword is optional in most cases (such as when declaring an alias for column names, constants, function calls, and simple unary operator expressions). In cases where the declared alias is a reserved SQL keyword, the *output\_name* must be enclosed in double quotes to avoid ambiguity.
+
+An *expression* in the `SELECT` list can be a constant value, a column reference, an operator invocation, a function call, an aggregate expression, a window expression, a scalar subquery, and so on. There are a number of constructs that can be classified as an expression but do not follow any general syntax rules.
+
+Instead of an expression, `*` can be written in the output list as a shorthand for all the columns of the selected rows. Also, you can write ` table_name.*` as a shorthand for the columns coming from just that table.
+
+**The FROM Clause**
+
+The `FROM` clause specifies one or more source tables for the `SELECT`. If multiple sources are specified, the result is the Cartesian product (cross join) of all the sources. But usually qualification conditions are added to restrict the returned rows to a small subset of the Cartesian product. The `FROM` clause can contain the following elements:
+
+ *table\_name*
+The name (optionally schema-qualified) of an existing table or view. If `ONLY` is specified, only that table is scanned. If `ONLY` is not specified, the table and all its descendant tables (if any) are scanned.
+
+ *alias*
+A substitute name for the `FROM` item containing the alias. An alias is used for brevity or to eliminate ambiguity for self-joins (where the same table is scanned multiple times). When an alias is provided, it completely hides the actual name of the table or function; for example given `FROM foo AS f`, the remainder of the `SELECT` must refer to this `FROM` item as `f` not `foo`. If an alias is written, a column alias list can also be written to provide substitute names for one or more columns of the table.
+
+ *select*
+A sub-`SELECT` can appear in the `FROM` clause. This acts as though its output were created as a temporary table for the duration of this single `SELECT` command. Note that the sub-`SELECT` must be surrounded by parentheses, and an alias must be provided for it. A `VALUES` command can also be used here. See "Non-standard Clauses" in the [Compatibility](#topic1__section19) section for limitations of using correlated sub-selects in HAWQ.
+
+ *function\_name*
+Function calls can appear in the `FROM` clause. (This is especially useful for functions that return result sets, but any function can be used.) This acts as though its output were created as a temporary table for the duration of this single `SELECT` command. An alias may also be used. If an alias is written, a column alias list can also be written to provide substitute names for one or more attributes of the function's composite return type. If the function has been defined as returning the record data type, then an alias or the key word `AS` must be present, followed by a column definition list in the form `( column_name data_type [, ... ] )`. The column definition list must match the actual number and types of columns returned by the function.
+
+ *join\_type*
+One of:
+
+- **\[INNER\] JOIN**
+- **LEFT \[OUTER\] JOIN**
+- **RIGHT \[OUTER\] JOIN**
+- **FULL \[OUTER\] JOIN**
+- **CROSS JOIN**
+
+For the `INNER` and `OUTER` join types, a join condition must be specified, namely exactly one of `NATURAL`, `ON join_condition `, or `USING ( join_column [, ...])`. See below for the meaning. For `CROSS JOIN`, none of these clauses may appear.
+
+A `JOIN` clause combines two `FROM` items. Use parentheses if necessary to determine the order of nesting. In the absence of parentheses, `JOIN`s nest left-to-right. In any case `JOIN` binds more tightly than the commas separating `FROM` items.
+
+`CROSS JOIN` and `INNER JOIN` produce a simple Cartesian product, the same result as you get from listing the two items at the top level of `FROM`, but restricted by the join condition (if any). `CROSS JOIN` is equivalent to `INNER JOIN ON` `(TRUE)`, that is, no rows are removed by qualification. These join types are just a notational convenience, since they do nothing you could not do with plain `FROM` and `WHERE`.
+
+`LEFT OUTER JOIN` returns all rows in the qualified Cartesian product (i.e., all combined rows that pass its join condition), plus one copy of each row in the left-hand table for which there was no right-hand row that passed the join condition. This left-hand row is extended to the full width of the joined table by inserting null values for the right-hand columns. Note that only the `JOIN` clause's own condition is considered while deciding which rows have matches. Outer conditions are applied afterwards.
+
+Conversely, `RIGHT OUTER JOIN` returns all the joined rows, plus one row for each unmatched right-hand row (extended with nulls on the left). This is just a notational convenience, since you could convert it to a `LEFT OUTER JOIN` by switching the left and right inputs.
+
+`FULL OUTER JOIN` returns all the joined rows, plus one row for each unmatched left-hand row (extended with nulls on the right), plus one row for each unmatched right-hand row (extended with nulls on the left).
+
+ON *join\_condition*
+*join\_condition* is an expression resulting in a value of type `boolean` (similar to a `WHERE` clause) that specifies which rows in a join are considered to match.
+
+USING (*join\_column* \[, ...\])
+A clause of the form `USING ( a, b, ... )` is shorthand for `ON left_table.a = right_table.a AND left_table.b = right_table.b ... `. Also, `USING` implies that only one of each pair of equivalent columns will be included in the join output, not both.
+
+NATURAL
+`NATURAL` is shorthand for a `USING` list that mentions all columns in the two tables that have the same names.
+
+**The WHERE Clause**
+
+The optional `WHERE` clause has the general form:
+
+``` pre
+WHERE condition
+
+```
+
+where *condition* is any expression that evaluates to a result of type `boolean`. Any row that does not satisfy this condition will be eliminated from the output. A row satisfies the condition if it returns true when the actual row values are substituted for any variable references.
+
+**The GROUP BY Clause**
+
+The optional `GROUP BY` clause has the general form:
+
+``` pre
+GROUP BY grouping_element [, ...]
+```
+
+where *grouping\_element* can be one of:
+
+``` pre
+()
+expression
+ROLLUP (expression [,...])
+CUBE (expression [,...])
+GROUPING SETS ((grouping_element [, ...]))
+```
+
+`GROUP BY` will condense into a single row all selected rows that share the same values for the grouped expressions. *expression* can be an input column name, or the name or ordinal number of an output column (`SELECT` list item), or an arbitrary expression formed from input-column values. In case of ambiguity, a `GROUP BY` name will be interpreted as an input-column name rather than an output column name.
+
+Aggregate functions, if any are used, are computed across all rows making up each group, producing a separate value for each group (whereas without `GROUP BY`, an aggregate produces a single value computed across all the selected rows). When `GROUP BY` is present, it is not valid for the `SELECT` list expressions to refer to ungrouped columns except within aggregate functions, since there would be more than one possible value to return for an ungrouped column.
+
+HAWQ has the following additional OLAP grouping extensions (often referred to as *supergroups*):
+
+ROLLUP
+A `ROLLUP` grouping is an extension to the `GROUP BY` clause that creates aggregate subtotals that roll up from the most detailed level to a grand total, following a list of grouping columns (or expressions). `ROLLUP` takes an ordered list of grouping columns, calculates the standard aggregate values specified in the `GROUP BY` clause, then creates progressively higher-level subtotals, moving from right to left through the list. Finally, it creates a grand total. A `ROLLUP` grouping can be thought of as a series of grouping sets. For example:
+
+``` pre
+GROUP BY ROLLUP (a,b,c)
+```
+
+is equivalent to:
+
+``` pre
+GROUP BY GROUPING SETS( (a,b,c), (a,b), (a), () )
+```
+
+Notice that the *n* elements of a `ROLLUP` translate to *n*+1 grouping sets. Also, the order in which the grouping expressions are specified is significant in a `ROLLUP`.
+
+CUBE
+A `CUBE` grouping is an extension to the `GROUP BY` clause that creates subtotals for all of the possible combinations of the given list of grouping columns (or expressions). In terms of multidimensional analysis, `CUBE` generates all the subtotals that could be calculated for a data cube with the specified dimensions. For example:
+
+``` pre
+GROUP BY CUBE (a,b,c)
+```
+
+is equivalent to:
+
+``` pre
+GROUP BY GROUPING SETS( (a,b,c), (a,b), (a,c), (b,c), (a),
+(b), (c), () )
+```
+
+Notice that *n* elements of a `CUBE` translate to 2n grouping sets. Consider using `CUBE` in any situation requiring cross-tabular reports. `CUBE` is typically most suitable in queries that use columns from multiple dimensions rather than columns representing different levels of a single dimension. For instance, a commonly requested cross-tabulation might need subtotals for all the combinations of month, state, and product.
+
+GROUPING SETS
+You can selectively specify the set of groups that you want to create using a `GROUPING SETS` expression within a `GROUP BY` clause. This allows precise specification across multiple dimensions without computing a whole `ROLLUP` or `CUBE`. For example:
+
+``` pre
+GROUP BY GROUPING SETS( (a,c), (a,b) )
+```
+
+If using the grouping extension clauses `ROLLUP`, `CUBE`, or `GROUPING SETS`, two challenges arise. First, how do you determine which result rows are subtotals, and then the exact level of aggregation for a given subtotal. Or, how do you differentiate between result rows that contain both stored `NULL` values and "NULL" values created by the `ROLLUP` or `CUBE`. Secondly, when duplicate grouping sets are specified in the `GROUP BY` clause, how do you determine which result rows are duplicates? There are two additional grouping functions you can use in the `SELECT` list to help with this:
+
+- **grouping(column \[, ...\])** — The `grouping` function can be applied to one or more grouping attributes to distinguish super-aggregated rows from regular grouped rows. This can be helpful in distinguishing a "NULL" representing the set of all values in a super-aggregated row from a `NULL` value in a regular row. Each argument in this function produces a bit — either `1` or `0`, where `1` means the result row is super-aggregated, and `0` means the result row is from a regular grouping. The `grouping` function returns an integer by treating these bits as a binary number and then converting it to a base-10 integer.
+- **group\_id()** — For grouping extension queries that contain duplicate grouping sets, the `group_id` function is used to identify duplicate rows in the output. All *unique* grouping set output rows will have a group\_id value of 0. For each duplicate grouping set detected, the `group_id` function assigns a group\_id number greater than 0. All output rows in a particular duplicate grouping set are identified by the same group\_id number.
+
+**The WINDOW Clause**
+
+The `WINDOW` clause is used to define a window that can be used in the `OVER()` expression of a window function such as `rank` or `avg`. For example:
+
+``` pre
+SELECT vendor, rank() OVER (mywindow) FROM sale
+GROUP BY vendor
+WINDOW mywindow AS (ORDER BY sum(prc*qty));
+```
+
+A `WINDOW` clause has this general form:
+
+``` pre
+WINDOW window_name AS (window_specification)
+```
+
+where *window\_specification* can be:
+
+``` pre
+[window_name]
+[PARTITION BY expression [, ...]]
+[ORDER BY expression [ASC | DESC | USING operator] [, ...]
+ [{RANGE | ROWS}
+ { UNBOUNDED PRECEDING
+ | expression PRECEDING
+ | CURRENT ROW
+ | BETWEEN window_frame_bound AND window_frame_bound }]]
+ where window_frame_bound can be one of:
+ UNBOUNDED PRECEDING
+ expression PRECEDING
+ CURRENT ROW
+ expression FOLLOWING
+ UNBOUNDED FOLLOWING
+```
+
+ *window\_name*
+Gives a name to the window specification.
+
+PARTITION BY
+The `PARTITION BY` clause organizes the result set into logical groups based on the unique values of the specified expression. When used with window functions, the functions are applied to each partition independently. For example, if you follow `PARTITION BY` with a column name, the result set is partitioned by the distinct values of that column. If omitted, the entire result set is considered one partition.
+
+ORDER BY
+The `ORDER BY` clause defines how to sort the rows in each partition of the result set. If omitted, rows are returned in whatever order is most efficient and may vary.
+
+**Note:** Columns of data types that lack a coherent ordering, such as `time`, are not good candidates for use in the `ORDER BY` clause of a window specification. Time, with or without time zone, lacks a coherent ordering because addition and subtraction do not have the expected effects. For example, the following is not generally true: `x::time < x::time + '2 hour'::interval`
+
+ROWS | RANGE
+Use either a `ROWS` or `RANGE` clause to express the bounds of the window. The window bound can be one, many, or all rows of a partition. You can express the bound of the window either in terms of a range of data values offset from the value in the current row (`RANGE`), or in terms of the number of rows offset from the currentrow (`ROWS`). When using the `RANGE` clause, you must also use an `ORDER BY` clause. This is because the calculation performed to produce the window requires that the values be sorted. Additionally, the `ORDER BY` clause cannot contain more than one expression, and the expression must result in either a date or a numeric value. When using the `ROWS` or `RANGE` clauses, if you specify only a starting row, the current row is used as the last row in the window.
+
+**PRECEDING** — The `PRECEDING` clause defines the first row of the window using the current row as a reference point. The starting row is expressed in terms of the number of rows preceding the current row. For example, in the case of `ROWS` framing, 5 `PRECEDING` sets the window to start with the fifth row preceding the current row. In the case of `RANGE` framing, it sets the window to start with the first row whose ordering column value precedes that of the current row by 5 in the given order. If the specified order is ascending by date, this will be the first row within 5 days before the current row. `UNBOUNDED PRECEDING` sets the first row in the window to be the first row in the partition.
+
+**BETWEEN** — The `BETWEEN` clause defines the first and last row of the window, using the current row as a reference point. First and last rows are expressed in terms of the number of rows preceding and following the current row, respectively. For example, `BETWEEN 3 PRECEDING AND 5 FOLLOWING` sets the window to start with the third row preceding the current row, and end with the fifth row following the current row. Use `BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING` to set the first and last rows in the window to be the first and last row in the partition, respectively. This is equivalent to the default behavior if no `ROW` or `RANGE` clause is specified.
+
+**FOLLOWING** — The `FOLLOWING` clause defines the last row of the window using the current row as a reference point. The last row is expressed in terms of the number of rows following the current row. For example, in the case of `ROWS` framing, `5 FOLLOWING` sets the window to end with the fifth row following the current row. In the case of `RANGE` framing, it sets the window to end with the last row whose ordering column value follows that of the current row by 5 in the given order. If the specified order is ascending by date, this will be the last row within 5 days after the current row. Use `UNBOUNDED FOLLOWING` to set the last row in the window to be the last row in the partition.
+
+If you do not specify a `ROW` or a `RANGE` clause, the window bound starts with the first row in the partition (`UNBOUNDED PRECEDING`) and ends with the current row (`CURRENT ROW`) if `ORDER BY` is used. If an `ORDER BY` is not specified, the window starts with the first row in the partition (`UNBOUNDED PRECEDING`) and ends with last row in the partition (`UNBOUNDED FOLLOWING`).
+
+**The HAVING Clause**
+
+The optional `HAVING` clause has the general form:
+
+``` pre
+HAVING condition
+
+```
+
+where *condition* is the same as specified for the `WHERE` clause. `HAVING` eliminates group rows that do not satisfy the condition. `HAVING` is different from `WHERE`: `WHERE` filters individual rows before the application of `GROUP BY`, while `HAVING` filters group rows created by `GROUP BY`. Each column referenced in *condition* must unambiguously reference a grouping column, unless the reference appears within an aggregate function.
+
+The presence of `HAVING` turns a query into a grouped query even if there is no `GROUP BY` clause. This is the same as what happens when the query contains aggregate functions but no `GROUP BY` clause. All the selected rows are considered to form a single group, and the `SELECT` list and `HAVING` clause can only reference table columns from within aggregate functions. Such a query will emit a single row if the `HAVING` condition is true, zero rows if it is not true.
+
+**The UNION Clause**
+
+The `UNION` clause has this general form:
+
+``` pre
+ select_statement UNION [ALL] select_statement
+
+```
+
+where *select\_statement* is any `SELECT` statement without an `ORDER BY`, `LIMIT`, `FOR UPDATE`, or `FOR SHARE` clause. (`ORDER BY` and `LIMIT` can be attached to a subquery expression if it is enclosed in parentheses. Without parentheses, these clauses will be taken to apply to the result of the `UNION`, not to its right-hand input expression.)
+
+The `UNION` operator computes the set union of the rows returned by the involved `SELECT` statements. A row is in the set union of two result sets if it appears in at least one of the result sets. The two `SELECT` statements that represent the direct operands of the `UNION` must produce the same number of columns, and corresponding columns must be of compatible data types.
+
+The result of `UNION` does not contain any duplicate rows unless the `ALL` option is specified. `ALL` prevents elimination of duplicates. (Therefore, `UNION ALL` is usually significantly quicker than `UNION`; use `ALL` when you can.)
+
+Multiple `UNION` operators in the same `SELECT` statement are evaluated left to right, unless otherwise indicated by parentheses.
+
+Currently, `FOR UPDATE` and `FOR SHARE` may not be specified either for a `UNION` result or for any input of a `UNION`.
+
+**The INTERSECT Clause**
+
+The `INTERSECT` clause has this general form:
+
+``` pre
+ select_statement INTERSECT [ALL] select_statement
+
+```
+
+where *select\_statement* is any SELECT statement without an `ORDER BY`, `LIMIT`, `FOR UPDATE`, or `FOR SHARE` clause.
+
+The `INTERSECT` operator computes the set intersection of the rows returned by the involved `SELECT` statements. A row is in the intersection of two result sets if it appears in both result sets.
+
+The result of `INTERSECT` does not contain any duplicate rows unless the `ALL` option is specified. With `ALL`, a row that has *m* duplicates in the left table and *n* duplicates in the right table will appear min(*m*, *n*) times in the result set.
+
+Multiple `INTERSECT` operators in the same `SELECT` statement are evaluated left to right, unless parentheses dictate otherwise. `INTERSECT` binds more tightly than `UNION`. That is, `A UNION B INTERSECT C` will be read as `A UNION (B INTERSECT C)`.
+
+Currently, `FOR UPDATE` and `FOR SHARE` may not be specified either for an `INTERSECT` result or for any input of an `INTERSECT`.
+
+**The EXCEPT Clause**
+
+The `EXCEPT` clause has this general form:
+
+``` pre
+ select_statement EXCEPT [ALL] select_statement
+
+```
+
+where *select\_statement* is any `SELECT` statement without an `ORDER BY`, `LIMIT`, `FOR UPDATE`, or `FOR SHARE` clause.
+
+The `EXCEPT` operator computes the set of rows that are in the result of the left `SELECT` statement but not in the result of the right one.
+
+The result of `EXCEPT` does not contain any duplicate rows unless the `ALL` option is specified. With `ALL`, a row that has *m* duplicates in the left table and *n* duplicates in the right table will appear max(*m-n*,0) times in the result set.
+
+Multiple `EXCEPT` operators in the same `SELECT` statement are evaluated left to right unless parentheses dictate otherwise. `EXCEPT` binds at the same level as `UNION`.
+
+Currently, `FOR UPDATE` and `FOR SHARE` may not be specified either for an `EXCEPT` result or for any input of an `EXCEPT`.
+
+**The ORDER BY Clause**
+
+The optional `ORDER BY` clause has this general form:
+
+``` pre
+ORDER BY expression [ASC | DESC | USING operator] [, ...]
+```
+
+where *expression* can be the name or ordinal number of an output column (`SELECT` list item), or it can be an arbitrary expression formed from input-column values.
+
+The `ORDER BY` clause causes the result rows to be sorted according to the specified expressions. If two rows are equal according to the left-most expression, they are compared according to the next expression and so on. If they are equal according to all specified expressions, they are returned in an implementation-dependent order.
+
+The ordinal number refers to the ordinal (left-to-right) position of the result column. This feature makes it possible to define an ordering on the basis of a column that does not have a unique name. This is never absolutely necessary because it is always possible to assign a name to a result column using the `AS` clause.
+
+It is also possible to use arbitrary expressions in the `ORDER BY` clause, including columns that do not appear in the `SELECT` result list. Thus the following statement is valid:
+
+``` pre
+SELECT name FROM distributors ORDER BY code;
+```
+
+A limitation of this feature is that an `ORDER BY` clause applying to the result of a `UNION`, `INTERSECT`, or `EXCEPT` clause may only specify an output column name or number, not an expression.
+
+If an `ORDER BY` expression is a simple name that matches both a result column name and an input column name, `ORDER BY` will interpret it as the result column name. This is the opposite of the choice that `GROUP BY` will make in the same situation. This inconsistency is made to be compatible with the SQL standard.
+
+Optionally one may add the key word `ASC` (ascending) or `DESC` (descending) after any expression in the `ORDER BY` clause. If not specified, `ASC` is assumed by default. Alternatively, a specific ordering operator name may be specified in the `USING` clause. `ASC` is usually equivalent to `USING <` and `DESC` is usually equivalent to `USING >`. (But the creator of a user-defined data type can define exactly what the default sort ordering is, and it might correspond to operators with other names.)
+
+The null value sorts higher than any other value. In other words, with ascending sort order, null values sort at the end, and with descending sort order, null values sort at the beginning.
+
+Character-string data is sorted according to the locale-specific collation order that was established when the HAWQ system was initialized.
+
+**The DISTINCT Clause**
+
+If `DISTINCT` is specified, all duplicate rows are removed from the result set (one row is kept from each group of duplicates). `ALL` specifies the opposite: all rows are kept. `ALL` is the default.
+
+`DISTINCT ON ( expression [, ...] )` keeps only the first row of each set of rows where the given expressions evaluate to equal. The `DISTINCT ON` expressions are interpreted using the same rules as for `ORDER BY`. Note that the 'first row' of each set is unpredictable unless `ORDER BY` is used to ensure that the desired row appears first. For example:
+
+``` pre
+SELECT DISTINCT ON (location) location, time, report FROM
+weather_reports ORDER BY location, time DESC;
+```
+
+retrieves the most recent weather report for each location. But if we had not used `ORDER BY` to force descending order of time values for each location, we would have gotten a report from an unpredictable time for each location.
+
+The `DISTINCT ON` expression(s) must match the left-most `ORDER BY` expression(s). The `ORDER BY` clause will normally contain additional expression(s) that determine the desired precedence of rows within each `DISTINCT ON` group.
+
+**The LIMIT Clause**
+
+The `LIMIT` clause consists of two independent sub-clauses:
+
+``` pre
+LIMIT {count | ALL}
+OFFSET start
+
+```
+
+where *count* specifies the maximum number of rows to return, while *start* specifies the number of rows to skip before starting to return rows. When both are specified, start rows are skipped before starting to count the count rows to be returned.
+
+When using `LIMIT`, it is a good idea to use an `ORDER BY` clause that constrains the result rows into a unique order. Otherwise you will get an unpredictable subset of the query's rows. You may be asking for the tenth through twentieth rows, but tenth through twentieth in what ordering? You don't know what ordering unless you specify `ORDER BY`.
+
+The query planner takes `LIMIT` into account when generating a query plan, so you are very likely to get different plans (yielding different row orders) depending on what you use for `LIMIT` and `OFFSET`. Thus, using different `LIMIT/OFFSET` values to select different subsets of a query result will give inconsistent results unless you enforce a predictable result ordering with `ORDER BY`. This is not a defect; it is an inherent consequence of the fact that SQL does not promise to deliver the results of a query in any particular order unless `ORDER BY` is used to constrain the order.
+
+## Examples<a id="topic1__section18"></a>
+
+To join the table `films` with the table `distributors`:
+
+``` pre
+SELECT f.title, f.did, d.name, f.date_prod, f.kind FROM
+distributors d, films f WHERE f.did = d.did
+```
+
+To sum the column `length` of all films and group the results by `kind`:
+
+``` pre
+SELECT kind, sum(length) AS total FROM films GROUP BY kind;
+```
+
+To sum the column `length` of all films, group the results by `kind` and show those group totals that are less than 5 hours:
+
+``` pre
+SELECT kind, sum(length) AS total FROM films GROUP BY kind
+HAVING sum(length) < interval '5 hours';
+```
+
+Calculate the subtotals and grand totals of all sales for movie `kind` and `distributor`.
+
+``` pre
+SELECT kind, distributor, sum(prc*qty) FROM sales
+GROUP BY ROLLUP(kind, distributor)
+ORDER BY 1,2,3;
+```
+
+Calculate the rank of movie distributors based on total sales:
+
+``` pre
+SELECT distributor, sum(prc*qty),
+ rank() OVER (ORDER BY sum(prc*qty) DESC)
+FROM sale
+GROUP BY distributor ORDER BY 2 DESC;
+```
+
+The following two examples are identical ways of sorting the individual results according to the contents of the second column (`name`):
+
+``` pre
+SELECT * FROM distributors ORDER BY name;
+SELECT * FROM distributors ORDER BY 2;
+```
+
+The next example shows how to obtain the union of the tables `distributors` and `actors`, restricting the results to those that begin with the letter `W` in each table. Only distinct rows are wanted, so the key word `ALL` is omitted:
+
+``` pre
+SELECT distributors.name FROM distributors WHERE
+distributors.name LIKE 'W%' UNION SELECT actors.name FROM
+actors WHERE actors.name LIKE 'W%';
+```
+
+This example shows how to use a function in the `FROM` clause, both with and without a column definition list:
+
+``` pre
+CREATE FUNCTION distributors(int) RETURNS SETOF distributors
+AS $$ SELECT * FROM distributors WHERE did = $1; $$ LANGUAGE
+SQL;
+SELECT * FROM distributors(111);
+
+CREATE FUNCTION distributors_2(int) RETURNS SETOF record AS
+$$ SELECT * FROM distributors WHERE did = $1; $$ LANGUAGE
+SQL;
+SELECT * FROM distributors_2(111) AS (dist_id int, dist_name
+text);
+```
+
+## Compatibility<a id="topic1__section19"></a>
+
+The `SELECT` statement is compatible with the SQL standard, but there are some extensions and some missing features.
+
+**Omitted FROM Clauses**
+
+HAWQ allows you to omit the `FROM` clause. It has a straightforward use to compute the results of simple expressions. For example:
+
+``` pre
+SELECT 2+2;
+```
+
+Some other SQL databases cannot do this except by introducing a dummy one-row table from which to do the `SELECT`.
+
+Note that if a `FROM` clause is not specified, the query cannot reference any database tables. For compatibility with applications that rely on this behavior the *add\_missing\_from* configuration variable can be enabled.
+
+**The AS Key Word**
+
+In the SQL standard, the optional key word `AS` is just noise and can be omitted without affecting the meaning. The HAWQ parser requires this key word when renaming output columns because the type extensibility features lead to parsing ambiguities without it. `AS` is optional in `FROM` items, however.
+
+**Namespace Available to GROUP BY and ORDER BY**
+
+In the SQL-92 standard, an `ORDER BY` clause may only use result column names or numbers, while a `GROUP BY` clause may only use expressions based on input column names. HAWQ extends each of these clauses to allow the other choice as well (but it uses the standard's interpretation if there is ambiguity). HAWQ also allows both clauses to specify arbitrary expressions. Note that names appearing in an expression will always be taken as input-column names, not as result-column names.
+
+SQL:1999 and later use a slightly different definition which is not entirely upward compatible with SQL-92. In most cases, however, HAWQ will interpret an `ORDER BY` or `GROUP BY` expression the same way SQL:1999 does.
+
+**Nonstandard Clauses**
+
+The clauses `DISTINCT ON`, `LIMIT`, and `OFFSET` are not defined in the SQL standard.
+
+**Limited Use of STABLE and VOLATILE Functions**
+
+To prevent data from becoming out-of-sync across the segments in HAWQ, any function classified as `STABLE` or `VOLATILE` cannot be executed at the segment database level if it contains SQL or modifies the database in any way.
+
+## See Also<a id="topic1__section25"></a>
+
+[EXPLAIN](EXPLAIN.html)
diff --git a/reference/sql/SET-ROLE.html.md.erb b/reference/sql/SET-ROLE.html.md.erb
new file mode 100644
index 0000000..ea7194f
--- /dev/null
+++ b/reference/sql/SET-ROLE.html.md.erb
@@ -0,0 +1,68 @@
+---
+title: SET ROLE
+---
+
+Sets the current role identifier of the current session.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+SET [SESSION | LOCAL] ROLE rolename
+SET [SESSION | LOCAL] ROLE NONE
+RESET ROLE
+```
+
+## Description<a id="topic1__section3"></a>
+
+This command sets the current role identifier of the current SQL-session context to be *rolename*. The role name may be written as either an identifier or a string literal. After `SET ROLE`, permissions checking for SQL commands is carried out as though the named role were the one that had logged in originally.
+
+The specified *rolename* must be a role that the current session user is a member of. If the session user is a superuser, any role can be selected.
+
+The `NONE` and `RESET` forms reset the current role identifier to be the current session role identifier. These forms may be executed by any user.
+
+## Parameters<a id="topic1__section4"></a>
+
+SESSION
+Specifies that the command takes effect for the current session. This is the default.
+
+LOCAL
+Specifies that the command takes effect for only the current transaction. After `COMMIT` or `ROLLBACK`, the session-level setting takes effect again. Note that `SET LOCAL` will appear to have no effect if it is executed outside of a transaction.
+
+ *rolename*
+The name of a role to use for permissions checking in this session.
+
+NONE
+RESET
+Reset the current role identifier to be the current session role identifier (that of the role used to log in).
+
+## Notes<a id="topic1__section5"></a>
+
+Using this command, it is possible to either add privileges or restrict privileges. If the session user role has the `INHERITS` attribute, then it automatically has all the privileges of every role that it could `SET ROLE` to; in this case `SET ROLE` effectively drops all the privileges assigned directly to the session user and to the other roles it is a member of, leaving only the privileges available to the named role. On the other hand, if the session user role has the `NOINHERITS` attribute, `SET ROLE` drops the privileges assigned directly to the session user and instead acquires the privileges available to the named role.
+
+In particular, when a superuser chooses to `SET ROLE` to a non-superuser role, she loses her superuser privileges.
+
+`SET ROLE` has effects comparable to `SET SESSION AUTHORIZATION`, but the privilege checks involved are quite different. Also, `SET SESSION AUTHORIZATION` determines which roles are allowable for later `SET ROLE` commands, whereas changing roles with `SET ROLE` does not change the set of roles allowed to a later `SET ROLE`.
+
+## Examples<a id="topic1__section6"></a>
+
+``` pre
+SELECT SESSION_USER, CURRENT_USER;
+ session_user | current_user
+--------------+--------------
+ peter | peter
+
+SET ROLE 'paul';
+
+SELECT SESSION_USER, CURRENT_USER;
+ session_user | current_user
+--------------+--------------
+ peter | paul
+```
+
+## Compatibility<a id="topic1__section7"></a>
+
+HAWQ allows identifier syntax (*rolename*), while the SQL standard requires the role name to be written as a string literal. SQL does not allow this command during a transaction; HAWQ does not make this restriction. The `SESSION` and `LOCAL` modifiers are a HAWQ extension, as is the `RESET` syntax.
+
+## See Also<a id="topic1__section8"></a>
+
+[SET SESSION AUTHORIZATION](SET-SESSION-AUTHORIZATION.html)
diff --git a/reference/sql/SET-SESSION-AUTHORIZATION.html.md.erb b/reference/sql/SET-SESSION-AUTHORIZATION.html.md.erb
new file mode 100644
index 0000000..f8d1df4
--- /dev/null
+++ b/reference/sql/SET-SESSION-AUTHORIZATION.html.md.erb
@@ -0,0 +1,62 @@
+---
+title: SET SESSION AUTHORIZATION
+---
+
+Sets the session role identifier and the current role identifier of the current session.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+SET [SESSION | LOCAL] SESSION AUTHORIZATION rolename
+SET [SESSION | LOCAL] SESSION AUTHORIZATION DEFAULT
+RESET SESSION AUTHORIZATION
+```
+
+## Description<a id="topic1__section3"></a>
+
+This command sets the session role identifier and the current role identifier of the current SQL-session context to be `rolename` . The role name may be written as either an identifier or a string literal. Using this command, it is possible, for example, to temporarily become an unprivileged user and later switch back to being a superuser.
+
+The session role identifier is initially set to be the (possibly authenticated) role name provided by the client. The current role identifier is normally equal to the session user identifier, but may change temporarily in the context of setuid functions and similar mechanisms; it can also be changed by [SET ROLE](SET-ROLE.html). The current user identifier is relevant for permission checking.
+
+The session user identifier may be changed only if the initial session user (the authenticated user) had the superuser privilege. Otherwise, the command is accepted only if it specifies the authenticated user name.
+
+The `DEFAULT` and `RESET` forms reset the session and current user identifiers to be the originally authenticated user name. These forms may be executed by any user.
+
+## Parameters<a id="topic1__section4"></a>
+
+SESSION
+Specifies that the command takes effect for the current session. This is the default.
+
+LOCAL
+Specifies that the command takes effect for only the current transaction. After `COMMIT` or `ROLLBACK`, the session-level setting takes effect again. Note that `SET LOCAL` will appear to have no effect if it is executed outside of a transaction.
+
+ *rolename*
+The name of the role to assume.
+
+NONE
+RESET
+Reset the session and current role identifiers to be that of the role used to log in.
+
+## Examples<a id="topic1__section5"></a>
+
+``` pre
+SELECT SESSION_USER, CURRENT_USER;
+ session_user | current_user
+--------------+--------------
+ peter | peter
+
+SET SESSION AUTHORIZATION 'paul';
+
+SELECT SESSION_USER, CURRENT_USER;
+ session_user | current_user
+--------------+--------------
+ paul | paul
+```
+
+## Compatibility<a id="topic1__section6"></a>
+
+The SQL standard allows some other expressions to appear in place of the literal *rolename*, but these options are not important in practice. HAWQ allows identifier syntax (*rolename*), which SQL does not. SQL does not allow this command during a transaction; HAWQ does not make this restriction. The `SESSION` and `LOCAL` modifiers are a HAWQ extension, as is the `RESET` syntax.
+
+## See Also<a id="topic1__section7"></a>
+
+[SET ROLE](SET-ROLE.html)
diff --git a/reference/sql/SET.html.md.erb b/reference/sql/SET.html.md.erb
new file mode 100644
index 0000000..e2776c1
--- /dev/null
+++ b/reference/sql/SET.html.md.erb
@@ -0,0 +1,89 @@
+---
+title: SET
+---
+
+Changes the value of a HAWQ configuration parameter.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+SET [SESSION | LOCAL] configuration_parameter {TO | =} value | 'value' | DEFAULT}
+SET [SESSION | LOCAL] TIME ZONE {timezone | LOCAL | DEFAULT}
+```
+
+## Description<a id="topic1__section3"></a>
+
+The `SET` command changes server configuration parameters. Any configuration parameter classified as a *session* parameter can be changed on-the-fly with `SET`. See [About Server Configuration Parameters](../guc/guc_config.html#topic1). `SET` only affects the value used by the current session.
+
+If `SET` or `SET SESSION` is issued within a transaction that is later aborted, the effects of the `SET` command disappear when the transaction is rolled back. Once the surrounding transaction is committed, the effects will persist until the end of the session, unless overridden by another `SET`.
+
+The effects of `SET LOCAL` only last till the end of the current transaction, whether committed or not. A special case is `SET` followed by `SET LOCAL` within a single transaction: the `SET LOCAL` value will be seen until the end of the transaction, but afterwards (if the transaction is committed) the `SET` value will take effect.
+
+## Parameters<a id="topic1__section4"></a>
+
+SESSION
+Specifies that the command takes effect for the current session. This is the default.
+
+LOCAL
+Specifies that the command takes effect for only the current transaction. After `COMMIT` or `ROLLBACK`, the session-level setting takes effect again. Note that `SET LOCAL` will appear to have no effect if it is executed outside of a transaction.
+
+ *configuration\_parameter*
+The name of a HAWQ configuration parameter. Only parameters classified as *session* can be changed with `SET`. See [About Server Configuration Parameters](../guc/guc_config.html#topic1).
+
+ *value*
+New value of parameter. Values can be specified as string constants, identifiers, numbers, or comma-separated lists of these. `DEFAULT` can be used to specify resetting the parameter to its default value. If specifying memory sizing or time units, enclose the value in single quotes.
+
+TIME ZONE
+`SET TIME ZONE` value is an alias for `SET timezone TO value `. The syntax `SET TIME ZONE` allows special syntax for the time zone specification. Here are examples of valid values:
+
+`'PST8PDT'`
+
+`'Europe/Rome'`
+
+`-7` (time zone 7 hours west from UTC)
+
+`INTERVAL '-08:00' HOUR TO MINUTE` (time zone 8 hours west from UTC).
+
+LOCAL,
+DEFAULT
+Set the time zone to your local time zone (the one that the server's operating system defaults to).
+
+## Examples<a id="topic1__section5"></a>
+
+Set the schema search path:
+
+``` pre
+SET search_path TO my_schema, public;
+```
+
+Increase work memory to 200 MB:
+
+``` pre
+SET work_mem TO '200MB';
+```
+
+Set the style of date to traditional POSTGRES with "day before month" input convention:
+
+``` pre
+SET datestyle TO postgres, dmy;
+```
+
+Set the time zone for San Mateo, California (Pacific Time):
+
+``` pre
+SET TIME ZONE 'PST8PDT';
+```
+
+Set the time zone for Italy:
+
+``` pre
+SET TIME ZONE 'Europe/Rome';
+```
+
+## Compatibility<a id="topic1__section6"></a>
+
+`SET TIME ZONE` extends syntax defined in the SQL standard. The standard allows only numeric time zone offsets while HAWQ allows more flexible time-zone specifications. All other `SET` features are HAWQ extensions.
+
+## See Also<a id="topic1__section7"></a>
+
+[RESET](RESET.html), [SHOW](SHOW.html)
diff --git a/reference/sql/SHOW.html.md.erb b/reference/sql/SHOW.html.md.erb
new file mode 100644
index 0000000..2ff899a
--- /dev/null
+++ b/reference/sql/SHOW.html.md.erb
@@ -0,0 +1,47 @@
+---
+title: SHOW
+---
+
+Shows the value of a system configuration parameter.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+SHOW configuration_parameter
+
+SHOW ALL
+```
+
+## Description<a id="topic1__section3"></a>
+
+`SHOW` displays the current settings of HAWQ system configuration parameters. These parameters can be set using the `SET` statement, or by editing the `hawq-site.xml` configuration file of the HAWQ master. Note that some parameters viewable by `SHOW` are read-only — their values can be viewed but not set. See [About Server Configuration Parameters](../guc/guc_config.html#topic1).
+
+## Parameters<a id="topic1__section4"></a>
+
+ *configuration\_parameter*
+The name of a system configuration parameter.
+
+ALL
+Shows the current value of all configuration parameters.
+
+## Examples<a id="topic1__section5"></a>
+
+Show the current setting of the parameter `search_path`:
+
+``` pre
+SHOW search_path;
+```
+
+Show the current setting of all parameters:
+
+``` pre
+SHOW ALL;
+```
+
+## Compatibility<a id="topic1__section6"></a>
+
+`SHOW` is a HAWQ extension.
+
+## See Also<a id="topic1__section7"></a>
+
+[SET](SET.html), [RESET](RESET.html)
diff --git a/reference/sql/TRUNCATE.html.md.erb b/reference/sql/TRUNCATE.html.md.erb
new file mode 100644
index 0000000..057db31
--- /dev/null
+++ b/reference/sql/TRUNCATE.html.md.erb
@@ -0,0 +1,52 @@
+---
+title: TRUNCATE
+---
+
+Empties a table of all rows.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+TRUNCATE [TABLE] name [, ...] [CASCADE | RESTRICT]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`TRUNCATE` quickly removes all rows from a table or set of tables.This is most useful on large tables.
+
+## Parameters<a id="topic1__section4"></a>
+
+ *name*
+The name (optionally schema-qualified) of a table to be truncated.
+
+CASCADE
+Since this key word applies to foreign key references (which are not supported in HAWQ) it has no effect.
+
+RESTRICT
+Since this key word applies to foreign key references (which are not supported in HAWQ) it has no effect.
+
+## Notes<a id="topic1__section5"></a>
+
+Only the owner of a table may `TRUNCATE` it. `TRUNCATE` will not perform the following:
+
+- Run any user-defined `ON DELETE` triggers that might exist for the tables.
+
+ **Note:** HAWQ does not support user-defined triggers.
+
+- Truncate any tables that inherit from the named table. Only the named table is truncated, not its child tables.
+
+## Examples<a id="topic1__section6"></a>
+
+Empty the table `films`:
+
+``` pre
+TRUNCATE films;
+```
+
+## Compatibility<a id="topic1__section7"></a>
+
+There is no `TRUNCATE` command in the SQL standard.
+
+## See Also<a id="topic1__section8"></a>
+
+[DROP TABLE](DROP-TABLE.html)
diff --git a/reference/sql/VACUUM.html.md.erb b/reference/sql/VACUUM.html.md.erb
new file mode 100644
index 0000000..7e0a8bb
--- /dev/null
+++ b/reference/sql/VACUUM.html.md.erb
@@ -0,0 +1,92 @@
+---
+title: VACUUM
+---
+
+Garbage-collects and optionally analyzes a database.
+
+## Synopsis<a id="topic1__section2"></a>
+
+``` pre
+VACUUM [FULL] [FREEZE] [VERBOSE] [table]
+VACUUM [FULL] [FREEZE] [VERBOSE] ANALYZE
+ [table [(column [, ...] )]]
+```
+
+## Description<a id="topic1__section3"></a>
+
+`VACUUM` reclaims storage occupied by deleted tuples. In normal HAWQ operation, tuples that are deleted or obsoleted by an update are not physically removed from their table; they remain present on disk until a `VACUUM` is done. Therefore it is necessary to do `VACUUM` periodically, especially on frequently-updated catalog tables. `VACUUM` has no effect on a normal HAWQ table, since the delete or update operations are not supported on normal HAWQ table.
+
+With no parameter, `VACUUM` processes every table in the current database. With a parameter, `VACUUM` processes only that table. `VACUUM ANALYZE` performs a `VACUUM` and then an `ANALYZE` for each selected table. This is a handy combination form for routine maintenance scripts. See [ANALYZE](ANALYZE.html) for more details about its processing.
+
+Plain `VACUUM` (without `FULL`) simply reclaims space and makes it available for re-use. This form of the command can operate in parallel with normal reading and writing of the table, as an exclusive lock is not obtained. `VACUUM FULL` does more extensive processing, including moving of tuples across blocks to try to compact the table to the minimum number of disk blocks. This form is much slower and requires an exclusive lock on each table while it is being processed.
+
+**Outputs**
+
+When `VERBOSE` is specified, `VACUUM` emits progress messages to indicate which table is currently being processed. Various statistics about the tables are printed as well.
+
+## Parameters<a id="topic1__section5"></a>
+
+FULL
+Selects a full vacuum, which may reclaim more space but takes much longer and exclusively locks the table.
+
+**Note:** A VACUUM FULL is not recommended in HAWQ. See [Notes](#topic1__section6).
+
+FREEZE
+Specifying `FREEZE` is equivalent to performing `VACUUM` with the `vacuum_freeze_min_age` server configuration parameter set to zero. The `FREEZE` option is deprecated and will be removed in a future release. Set the parameter by using `hawq config` or by modifying your `hawq-site.xml` configuration file instead.
+
+VERBOSE
+Prints a detailed vacuum activity report for each table.
+
+ANALYZE
+Updates statistics used by the planner to determine the most efficient way to execute a query.
+
+ *table*
+The name (optionally schema-qualified) of a specific table to vacuum. Defaults to all tables in the current database.
+
+ *column*
+The name of a specific column to analyze. Defaults to all columns.
+
+## Notes<a id="topic1__section6"></a>
+
+`VACUUM` cannot be executed inside a transaction block.
+
+A recommended practice is to vacuum active production databases frequently (at least nightly), in order to remove expired rows. After adding or deleting a large number of rows, it may be a good idea to issue a `VACUUM ANALYZE` command for the affected table. This will update the system catalogs with the results of all recent changes, and allow the HAWQ query planner to make better choices in planning queries.
+
+`VACUUM` causes a substantial increase in I/O traffic, which can cause poor performance for other active sessions. Therefore, it is advisable to vacuum the database at low usage times. The `auto vacuum` daemon feature, that automates the execution of `VACUUM` and `ANALYZE` commands is currently disabled in HAWQ.
+
+Expired rows are held in what is called the *free space map*. The free space map must be sized large enough to cover the dead rows of all tables in your database. If not sized large enough, space occupied by dead rows that overflow the free space map cannot be reclaimed by a regular `VACUUM` command.
+
+`VACUUM FULL` will reclaim all expired row space, but is a very expensive operation and may take an unacceptably long time to finish on large, distributed HAWQ tables. If you do get into a situation where the free space map has overflowed, it may be more timely to recreate the table with a `CREATE TABLE AS` statement and drop the old table.
+
+`VACUUM FULL` is not recommended in HAWQ. It is best to size the free space map appropriately. The free space map is configured with the following server configuration parameters:
+
+- `max_fsm_pages`
+- `max_fsm_relations`
+
+## Examples<a id="topic1__section7"></a>
+
+Vacuum all tables in the current database:
+
+``` pre
+VACUUM;
+```
+
+Vacuum a specific table only:
+
+``` pre
+VACUUM mytable;
+```
+
+Vacuum all tables in the current database and collect statistics for the query planner:
+
+``` pre
+VACUUM ANALYZE;
+```
+
+## Compatibility<a id="topic1__section8"></a>
+
+There is no `VACUUM` statement in the SQL standard.
+
+## See Also<a id="topic1__section9"></a>
+
+[ANALYZE](ANALYZE.html)
diff --git a/reference/toolkit/hawq_toolkit.html.md.erb b/reference/toolkit/hawq_toolkit.html.md.erb
new file mode 100644
index 0000000..02b133e
--- /dev/null
+++ b/reference/toolkit/hawq_toolkit.html.md.erb
@@ -0,0 +1,263 @@
+---
+title: The hawq_toolkit Administrative Schema
+---
+
+This section provides a reference on the `hawq_toolkit` administrative schema.
+
+HAWQ provides an administrative schema called `hawq_toolkit` that you can use to query the system catalogs, log files, and operating environment for system status information. The `hawq_toolkit` schema contains a number of views that you can access using SQL commands. The `hawq_toolkit` schema is accessible to all database users, although some objects may require superuser permissions.
+
+This documentation describes the most useful views in `hawq_toolkit`. You may notice other objects (views, functions, and external tables) within the `hawq_toolkit` schema that are not described in this documentation (these are supporting objects to the views described in this section).
+
+**Warning:** Do not change database objects in the `hawq_toolkit` schema. Do not create database objects in the schema. Changes to objects in the schema might affect the accuracy of administrative information returned by schema objects.
+
+## Checking for Tables that Need Routine Maintenance<a id="topic2"></a>
+
+The following views can help identify tables that need routine table maintenance (`VACUUM` and/or `ANALYZE`).
+
+- [hawq\_stats\_missing](#topic4)
+
+The `VACUUM` command reclaims disk space occupied by deleted or obsolete rows. Because of the MVCC transaction concurrency model used in HAWQ, data rows that are deleted or updated still occupy physical space on disk even though they are not visible to any new transactions. Expired rows increase table size on disk and eventually slow down scans of the table.
+
+**Note:** VACUUM FULL is not recommended in HAWQ. See [VACUUM](../sql/VACUUM.html#topic1).
+
+The `ANALYZE` command collects column-level statistics needed by the query optimizer. HAWQ uses a cost-based query optimizer that relies on database statistics. Accurate statistics allow the query optimizer to better estimate selectivity and the number of rows retrieved by a query operation in order to choose the most efficient query plan.
+
+### hawq\_stats\_missing<a id="topic4"></a>
+
+This view shows tables that do not have statistics and therefore may require an `ANALYZE` be run on the table.
+
+<a id="topic4__ie194266"></a>
+
+<span class="tablecap">Table 1. hawq\_stats\_missing view</span>
+
+| Column | Description |
+|-----------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| smischema | Schema name. |
+| smitable | Table name. |
+| smisize | Does this table have statistics? False if the table does not have row count and row sizing statistics recorded in the system catalog, which may indicate that the table needs to be analyzed. This will also be false if the table does not contain any rows. For example, the parent tables of partitioned tables are always empty and will always return a false result. |
+| smicols | Number of columns in the table. |
+| smirecs | Number of rows in the table. |
+
+
+## Viewing HAWQ Server Log Files<a id="topic16"></a>
+
+Each component of a HAWQ system (master, standby master, and segments) keeps its own server log files. The `hawq_log_*` family of views allows you to issue SQL queries against the server log files to find particular entries of interest. The use of these views requires superuser permissions.
+
+- [hawq\_log\_command\_timings](#topic17)
+- [hawq\_log\_master\_concise](#topic19)
+
+### hawq\_log\_command\_timings<a id="topic17"></a>
+
+This view uses an external table to read the log files on the master and report the execution time of SQL commands executed in a database session. The use of this view requires superuser permissions.
+
+<a id="topic17__ie176169"></a>
+
+<span class="tablecap">Table 2. hawq\_log\_command\_timings view</span>
+
+| Column | Description |
+|-------------|------------------------------------------------------------|
+| logsession | The session identifier (prefixed with "con"). |
+| logcmdcount | The command number within a session (prefixed with "cmd"). |
+| logdatabase | The name of the database. |
+| loguser | The name of the database user. |
+| logpid | The process id (prefixed with "p"). |
+| logtimemin | The time of the first log message for this command. |
+| logtimemax | The time of the last log message for this command. |
+| logduration | Statement duration from start to end time. |
+
+
+### hawq\_log\_master\_concise<a id="topic19"></a>
+
+This view uses an external table to read a subset of the log fields from the master log file. The use of this view requires superuser permissions.
+
+<a id="topic19__ie177543"></a>
+
+<span class="tablecap">Table 3. hawq\_log\_master\_concise view</span>
+
+| Column | Description |
+|-------------|------------------------------------------------------------|
+| logtime | The timestamp of the log message. |
+| logdatabase | The name of the database. |
+| logsession | The session identifier (prefixed with "con"). |
+| logcmdcount | The command number within a session (prefixed with "cmd"). |
+| logseverity | The severity level for the record. |
+| logmessage | Log or error message text. |
+
+
+## Checking Database Object Sizes and Disk Space<a id="topic38"></a>
+
+The `hawq_size_*` family of views can be used to determine the disk space usage for a distributed HAWQ, schema, table, or index. The following views calculate the total size of an object across all segments.
+
+- [hawq\_size\_of\_all\_table\_indexes](#topic39)
+- [hawq\_size\_of\_database](#topic40)
+- [hawq\_size\_of\_index](#topic41)
+- [hawq\_size\_of\_partition\_and\_indexes\_disk](#topic42)
+- [hawq\_size\_of\_schema\_disk](#topic43)
+- [hawq\_size\_of\_table\_and\_indexes\_disk](#topic44)
+- [hawq\_size\_of\_table\_and\_indexes\_licensing](#topic45)
+- [hawq\_size\_of\_table\_disk](#topic46)
+- [hawq\_size\_of\_table\_uncompressed](#topic47)
+
+The table and index sizing views list the relation by object ID (not by name). To check the size of a table or index by name, you must look up the relation name (`relname`) in the `pg_class` table. For example:
+
+``` pre
+SELECT relname as name, sotdsize as size, sotdtoastsize as
+toast, sotdadditionalsize as other
+FROM gp_size_of_table_disk as sotd, pg_class
+WHERE sotd.sotdoid=pg_class.oid ORDER BY relname;
+```
+
+### hawq\_size\_of\_all\_table\_indexes<a id="topic39"></a>
+
+This view shows the total size of all indexes for a table. This view is accessible to all users, however non-superusers will only be able to see relations that they have permission to access.
+
+<a id="topic39__ie181657"></a>
+
+<span class="tablecap">Table 4. hawq\_size\_of\_all\_table\_indexes view</span>
+
+| Column | Description |
+|-----------------|----------------------------------------------|
+| soatioid | The object ID of the table |
+| soatisize | The total size of all table indexes in bytes |
+| soatischemaname | The schema name |
+| soatitablename | The table name |
+
+
+### hawq\_size\_of\_database<a id="topic40"></a>
+
+This view shows the total size of a database. This view is accessible to all users, however non-superusers will only be able to see databases that they have permission to access.
+
+<a id="topic40__ie181758"></a>
+
+<span class="tablecap">Table 5. hawq\_size\_of\_database view</span>
+
+| Column | Description |
+|-------------|-----------------------------------|
+| sodddatname | The name of the database |
+| sodddatsize | The size of the database in bytes |
+
+
+### hawq\_size\_of\_index<a id="topic41"></a>
+
+This view shows the total size of an index. This view is accessible to all users, however non-superusers will only be able to see relations that they have permission to access.
+
+<a id="topic41__ie181709"></a>
+
+<span class="tablecap">Table 6. hawq\_size\_of\_index view</span>
+
+| Column | Description |
+|--------------------|-------------------------------------------------------|
+| soioid | The object ID of the index |
+| soitableoid | The object ID of the table to which the index belongs |
+| soisize | The size of the index in bytes |
+| soiindexschemaname | The name of the index schema |
+| soiindexname | The name of the index |
+| soitableschemaname | The name of the table schema |
+| soitablename | The name of the table |
+
+
+### hawq\_size\_of\_partition\_and\_indexes\_disk<a id="topic42"></a>
+
+This view shows the size on disk of partitioned child tables and their indexes. This view is accessible to all users, however non-superusers will only be able to see relations that they have permission to access.
+
+<a id="topic42__ie181803"></a>
+
+<span class="tablecap">Table 7. hawq\_size\_of\_partition\_and\_indexes\_disk view</span>
+
+| Column | Description |
+|----------------------------|-------------------------------------------------|
+| sopaidparentoid | The object ID of the parent table |
+| sopaidpartitionoid | The object ID of the partition table |
+| sopaidpartitiontablesize | The partition table size in bytes |
+| sopaidpartitionindexessize | The total size of all indexes on this partition |
+| Sopaidparentschemaname | The name of the parent schema |
+| Sopaidparenttablename | The name of the parent table |
+| Sopaidpartitionschemaname | The name of the partition schema |
+| sopaidpartitiontablename | The name of the partition table |
+
+
+### hawq\_size\_of\_schema\_disk<a id="topic43"></a>
+
+This view shows schema sizes for the public schema and the user-created schemas in the current database. This view is accessible to all users, however non-superusers will be able to see only the schemas that they have permission to access.
+
+<a id="topic43__ie183105"></a>
+
+<span class="tablecap">Table 8. hawq\_size\_of\_schema\_disk view</span>
+
+| Column | Description |
+|---------------------|--------------------------------------------------|
+| sosdnsp | The name of the schema |
+| sosdschematablesize | The total size of tables in the schema in bytes |
+| sosdschemaidxsize | The total size of indexes in the schema in bytes |
+
+
+### hawq\_size\_of\_table\_and\_indexes\_disk<a id="topic44"></a>
+
+This view shows the size on disk of tables and their indexes. This view is accessible to all users, however non-superusers will only be able to see relations that they have permission to access.
+
+<a id="topic44__ie183128"></a>
+
+<span class="tablecap">Table 9. hawq\_size\_of\_table\_and\_indexes\_disk view</span>
+
+| Column | Description |
+|------------------|--------------------------------------------|
+| sotaidoid | The object ID of the parent table |
+| sotaidtablesize | The disk size of the table |
+| sotaididxsize | The total size of all indexes on the table |
+| sotaidschemaname | The name of the schema |
+| sotaidtablename | The name of the table |
+
+
+### hawq\_size\_of\_table\_and\_indexes\_licensing<a id="topic45"></a>
+
+This view shows the total size of tables and their indexes for licensing purposes. The use of this view requires superuser permissions.
+
+<a id="topic45__ie181949"></a>
+
+<span class="tablecap">Table 10. hawq\_size\_of\_table\_and\_indexes\_licensing view</span>
+
+| Column | Description |
+|-----------------------------|---------------------------------------------------------------------------------------------|
+| sotailoid | The object ID of the table |
+| sotailtablesizedisk | The total disk size of the table |
+| sotailtablesizeuncompressed | If the table is a compressed append-only table, shows the uncompressed table size in bytes. |
+| sotailindexessize | The total size of all indexes in the table |
+| sotailschemaname | The schema name |
+| sotailtablename | The table name |
+
+
+### hawq\_size\_of\_table\_disk<a id="topic46"></a>
+
+This view shows the size of a table on disk. This view is accessible to all users, however non-superusers will only be able to see tables that they have permission to access
+
+<a id="topic46__ie183408"></a>
+
+<span class="tablecap">Table 11. hawq\_size\_of\_table\_disk view</span>
+
+| Column | Description |
+|--------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| sotdoid | The object ID of the table |
+| sotdsize | The size of the table in bytes. The size is only the main table size. The size does not include auxiliary objects such as oversized (toast) attributes, or additional storage objects for AO tables. |
+| sotdtoastsize | The size of the TOAST table (oversized attribute storage), if there is one. |
+| sotdadditionalsize | Reflects the segment and block directory table sizes for append-only (AO) tables. |
+| sotdschemaname | The schema name |
+| sotdtablename | The table name |
+
+
+### hawq\_size\_of\_table\_uncompressed<a id="topic47"></a>
+
+This view shows the uncompressed table size for append-only (AO) tables. Otherwise, the table size on disk is shown. The use of this view requires superuser permissions.
+
+<a id="topic47__ie183582"></a>
+
+<span class="tablecap">Table 12. hawq\_size\_of\_table\_uncompressed view</span>
+
+| Column | Description |
+|----------------|---------------------------------------------------------------------------------------------------------------|
+| sotuoid | The object ID of the table |
+| sotusize | The uncomressed size of the table in bytes if it is a compressed AO table. Otherwise, the table size on disk. |
+| sotuschemaname | The schema name |
+| sotutablename | The table name |
+
+
+
diff --git a/resourcemgmt/ConfigureResourceManagement.md b/resourcemgmt/ConfigureResourceManagement.md
index 25bb316..b7c20c0 100644
--- a/resourcemgmt/ConfigureResourceManagement.md
+++ b/resourcemgmt/ConfigureResourceManagement.md
@@ -19,7 +19,7 @@
To configure HAWQ to run without a global resource manager, add the following property configuration to your `hawq-site.xml` file:
-```
+``` xml
<property>
<name>hawq_global_rm_type</name>
<value>none</value>
@@ -40,7 +40,7 @@
In `hawq-site.xml`, add the following parameters:
-```
+``` xml
<property>
<name>hawq_rm_memory_limit_perseg</name>
<value>8GB</value>
@@ -81,14 +81,14 @@
The following configuration properties allow a user to control resource quotas without altering corresponding resource queues.
-- [hawq\_rm\_stmt\_vseg\_memory](/200/hawq/reference/guc/parameter_definitions.html)
-- [hawq\_rm\_stmt\_nvseg](/200/hawq/reference/guc/parameter_definitions.html)
+- [hawq\_rm\_stmt\_vseg\_memory](/20/reference/guc/parameter_definitions.html)
+- [hawq\_rm\_stmt\_nvseg](/20/reference/guc/parameter_definitions.html)
However, the changed resource quota for the virtual segment cannot exceed the resource queue’s maximum capacity in HAWQ.
In the following example, when executing the next query statement, the HAWQ resource manager will attempt to allocate 10 virtual segments and each segment has a 256MB memory quota.
-```
+``` sql
postgres=# set hawq_rm_stmt_vseg_memory='256mb';
SET
postgres=# set hawq_rm_stmt_nvseg=10;
@@ -117,4 +117,4 @@
- `hawq_rm_nvseg_perquery_limit` limits the maximum number of virtual segments that can be used for one statement execution on a cluster-wide level. The hash buckets defined in `default_hash_table_bucket_number` cannot exceed this number. The default value is 512.
- `default_hash_table_bucket_number` defines the number of buckets used by default when you create a hash table. When you query a hash table, the query's virtual segment resources are fixed and allocated based on the bucket number defined for the table. A best practice is to tune this configuration parameter after you expand the cluster.
-You can also limit the number of virtual segments used by queries when configuring your resource queues. \(See [CREATE RESOURCE QUEUE](/200/hawq/reference/sql/CREATE-RESOURCE-QUEUE.html).\) The global configuration parameters are a hard limit, however, and any limits set on the resource queue or on the statement-level cannot be larger than these limits set on the cluster-wide level.
+You can also limit the number of virtual segments used by queries when configuring your resource queues. \(See [CREATE RESOURCE QUEUE](/20/reference/sql/CREATE-RESOURCE-QUEUE.html).\) The global configuration parameters are a hard limit, however, and any limits set on the resource queue or on the statement-level cannot be larger than these limits set on the cluster-wide level.
diff --git a/resourcemgmt/ResourceManagerStatus.md b/resourcemgmt/ResourceManagerStatus.md
index 5dc8983..134848f 100644
--- a/resourcemgmt/ResourceManagerStatus.md
+++ b/resourcemgmt/ResourceManagerStatus.md
@@ -11,8 +11,11 @@
The following is an example query to obtain connection track status:
-```
+``` sql
postgres=# select * from dump_resource_manager_status(1);
+```
+
+``` pre
dump_resource_manager_status
----------------------------------------------------------------------------------------
Dump resource manager connection track status to /tmp/resource_manager_conntrack_status
@@ -21,7 +24,7 @@
The following output is an example of resource context \(connection track\) status.
-```
+``` pre
Number of free connection ids : 65535
Number of connection tracks having requests to handle : 0
Number of connection tracks having responses to send : 0SOCK(client=192.168.2.100:37396:time=2015-11-15-20:54:35.379006),
@@ -55,8 +58,11 @@
The following is a query to obtain resource queue status:
-```
+``` sql
postgres=# select * from dump_resource_manager_status(2);
+```
+
+``` pre
dump_resource_manager_status
-------------------------------------------------------------------------------------
Dump resource manager resource queue status to /tmp/resource_manager_resqueue_status
@@ -65,7 +71,7 @@
The possible output of resource queue status is shown as below.
-```
+``` pre
Maximum capacity of queue in global resource manager cluster 1.000000
Number of resource queues : 4
@@ -97,8 +103,11 @@
Use the following query to obtain the status of a HAWQ segment.
-```
+``` sql
postgres=# select * from dump_resource_manager_status(3);
+```
+
+``` pre
dump_resource_manager_status
-----------------------------------------------------------------------------------
Dump resource manager resource pool status to /tmp/resource_manager_respool_status
@@ -107,7 +116,7 @@
The following output shows the status of a HAWQ segment status. This example describes a host named `sdw1` having resource capacity 64GB memory and 16 vcore. It now has 64GB available resource ready for use and 16 containers are held.
-```
+``` pre
HOST_ID(id=0:hostname:sdw1)
HOST_INFO(FTSTotalMemoryMB=65536:FTSTotalCore=16:GRMTotalMemoryMB=0:GRMTotalCore=0)
HOST_AVAILABLITY(HAWQAvailable=true:GLOBAvailable=false)
diff --git a/resourcemgmt/ResourceQueues.md b/resourcemgmt/ResourceQueues.md
index f64c06a..f442497 100644
--- a/resourcemgmt/ResourceQueues.md
+++ b/resourcemgmt/ResourceQueues.md
@@ -30,7 +30,7 @@
**Enforcing Limits on Resources**
-You can configure HAWQ to enforce limits on resource usage by setting memory and CPU usage limits on both segments and resource queues. See [Configuring Segment Resource Capacity](ConfigureResourceManagement.html) and [Creating Resource Queues](ResourceQueues.html). For some best practices on designing and using resource queues in HAWQ, see [Best Practices for Managing Resources](/200/hawq/bestpractices/managing_resources_bestpractices.html).
+You can configure HAWQ to enforce limits on resource usage by setting memory and CPU usage limits on both segments and resource queues. See [Configuring Segment Resource Capacity](ConfigureResourceManagement.html) and [Creating Resource Queues](ResourceQueues.html). For some best practices on designing and using resource queues in HAWQ, see [Best Practices for Managing Resources](/20/bestpractices/managing_resources_bestpractices.html).
For a high-level overview of how resource management works in HAWQ, see [Managing Resources](HAWQResourceManagement.html).
@@ -42,7 +42,7 @@
You can configure this property in `hawq-site.xml`. The new maximum takes effect when HAWQ restarts. For example, the configuration below sets this value to 50.
-```
+``` xml
<property>
<name>hawq_rm_nresqueue_limit</name>
<value>50</value>
@@ -53,9 +53,11 @@
To check the currently configured limit, you can execute the following command:
-```
+``` sql
postgres=# show hawq_rm_nresqueue_limit;
+```
+``` pre
hawq_rm_nresqueue_limit
----------------------------------------------
128
@@ -66,7 +68,7 @@
Use CREATE RESOURCE QUEUE to create a new resource queue. Only a superuser can run this DDL statement.
-Creating a resource queue involves giving it a name, a parent, setting the CPU and memory limits for the queue, and optionally a limit to the number of active statements on the resource queue. See [CREATE RESOURCE QUEUE](/200/hawq/reference/sql/CREATE-RESOURCE-QUEUE.html).
+Creating a resource queue involves giving it a name, a parent, setting the CPU and memory limits for the queue, and optionally a limit to the number of active statements on the resource queue. See [CREATE RESOURCE QUEUE](/20/reference/sql/CREATE-RESOURCE-QUEUE.html).
**Note:** You can only associate roles and queries with leaf-level resource queues. Leaf-level resource queues are resource queues that do not have any children.
@@ -74,14 +76,14 @@
Create a resource queue as a child of `pg_root` with an active query limit of 20 and memory and core limits of 50%:
-```
+``` sql
CREATE RESOURCE QUEUE myqueue WITH (PARENT='pg_root', ACTIVE_STATEMENTS=20,
MEMORY_LIMIT_CLUSTER=50%, CORE_LIMIT_CLUSTER=50%);
```
Create a resource queue as a child of pg\_root with memory and CPU limits and a resource overcommit factor:
-```
+``` sql
CREATE RESOURCE QUEUE test_queue_1 WITH (PARENT='pg_root',
MEMORY_LIMIT_CLUSTER=50%, CORE_LIMIT_CLUSTER=50%, RESOURCE_OVERCOMMIT_FACTOR=2);
```
@@ -98,7 +100,7 @@
To prevent conflicts, HAWQ cancels by default all resource requests that are in conflict with the new resource queue definition. This behavior is controlled by the `hawq_rm_force_alterqueue_cancel_queued_request` server configuration parameter, which is by default set to true \(`on`\). If you set the server configuration parameter `hawq_rm_force_alterqueue_cancel_queued_request` to false, the actions specified in ALTER RESOURCE QUEUE are canceled if the resource manager finds at least one resource request that is in conflict with the new resource definitions supplied in the altering command.
-For more information, see [ALTER RESOURCE QUEUE](/200/hawq/reference/sql/ALTER-RESOURCE-QUEUE.html).
+For more information, see [ALTER RESOURCE QUEUE](/20/reference/sql/ALTER-RESOURCE-QUEUE.html).
**Note:** To change the roles \(users\) assigned to a resource queue, use the ALTER ROLE command.
@@ -106,14 +108,14 @@
Change the memory and core limit of a resource queue:
-```
+``` sql
ALTER RESOURCE QUEUE test_queue_1 WITH (MEMORY_LIMIT_CLUSTER=40%,
CORE_LIMIT_CLUSTER=40%);
```
Change the active statements maximum for the resource queue:
-```
+``` sql
ALTER RESOURCE QUEUE test_queue_1 WITH (ACTIVE_STATEMENTS=50);
```
@@ -129,13 +131,13 @@
Remove a role from a resource queue \(and move the role to the default resource queue, `pg_default`\):
-```
+``` sql
ALTER ROLE bob RESOURCE QUEUE NONE;
```
Remove the resource queue named `adhoc`:
-```
+``` sql
DROP RESOURCE QUEUE adhoc;
```
@@ -145,23 +147,28 @@
The following example shows the data selected from `pg_resqueue`.
-```
+``` sql
postgres=# SELECT rsqname,parentoid,activestats,memorylimit,corelimit,resovercommit,
allocpolicy,vsegresourcequota,nvsegupperlimit,nvseglowerlimit,nvsegupperlimitperseg,nvseglowerlimitperseg
FROM pg_resqueue WHERE rsqname='test_queue_1';
+```
+``` pre
rsqname | parentoid | activestats | memorylimit | corelimit | resovercommit | allocpolicy | vsegresourcequota | nvsegupperlimit | nvseglowerlimit |nvsegupperlimitperseg | nvseglowerlimitperseg
--------------+-----------+-------------+-------------+-----------+---------------+-------------+-------------------+-----------------+-----------------+-----------------------+-----------------------
test_queue_1 | 9800 | 100 | 50% | 50% | 2 | even | mem:128mb | 0 | 0 | 0 |1
```
-The query displays all the attributes and their values of the selected resource queue. See [CREATE RESOURCE QUEUE](/200/hawq/reference/sql/CREATE-RESOURCE-QUEUE.html) for a description of these attributes.
+The query displays all the attributes and their values of the selected resource queue. See [CREATE RESOURCE QUEUE](/20/reference/sql/CREATE-RESOURCE-QUEUE.html) for a description of these attributes.
You can also check the runtime status of existing resource queues by querying the `pg_resqueue_status` view:
-```
+``` sql
postgres=# select * from pg_resqueue_status;
+```
+
+``` pre
rsqname | segmem | segcore | segsize | segsizemax | inusemem | inusecore | rsqholders | rsqwaiters | paused
------------+--------+----------+---------+------------+----------+-----------+------------+------------+--------
pg_root | 128 | 0.125000 | 64 | 64 | 0 | 0.000000 | 0 | 0 | F
@@ -189,7 +196,7 @@
The following are some examples of creating and assigning a role to a resource queue:
-```
+``` sql
CREATE ROLE rmtest1 WITH LOGIN RESOURCE QUEUE pg_default;
ALTER ROLE rmtest1 RESOURCE QUEUE test_queue_1;
diff --git a/resourcemgmt/YARNIntegration.md b/resourcemgmt/YARNIntegration.md
index e142153..57aabed 100644
--- a/resourcemgmt/YARNIntegration.md
+++ b/resourcemgmt/YARNIntegration.md
@@ -30,7 +30,7 @@
The following example demonstrates how to configure CapacityScheduler as the YARN resource scheduler. In `yarn-site.xml`, use the following configuration to enable CapacityScheduler.
-```
+``` xml
<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
@@ -39,7 +39,7 @@
Then, define the queues in CapacityScheduler's configuration. In `capacity-scheduler.xml`, you could define the queues as follows:
-```
+``` xml
<property>
<name>yarn.scheduler.capacity.root.queues</name>
<value>mrque1,mrque2,hawqque</value>
@@ -51,7 +51,7 @@
In the following configuration within `capacity-scheduler.xml,` we configure the additional properties for the queues to control the capacity of each queue. The HAWQ resource queue can utilize 20% to a maximum of 80% resources of the whole cluster.
-```
+``` xml
<property>
<name>yarn.scheduler.capacity.hawqque.maximum-applications</name>
<value>1</value>
@@ -106,7 +106,7 @@
In HAWQ standalone mode, you can configure the resource capacity of individual segments as described in [Configuring Segment Resource Capacity](ConfigureResourceManagement.html). If you are using YARN to manage HAWQ resources, then you configure the resource capacity of segments by configuring YARN. We recommend that you configure all segments with identical resource capacity. In `yarn-site.xml`, set the following properties:
-```
+``` xml
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>4GB</value>
@@ -143,7 +143,7 @@
**Note:** If you are specifying 1GB or under for `yarn.scheduler.minimum-allocation-mb` in `yarn-site.xml`, then make sure that the property is an equal subdivision of 1GB. For example, 1024, 512.
-See [Handling Segment Resource Fragmentation](/200/hawq/troubleshooting/Troubleshooting.html) for general information on resource fragmentation.
+See [Handling Segment Resource Fragmentation](/20/troubleshooting/Troubleshooting.html) for general information on resource fragmentation.
## Enabling YARN Mode in HAWQ <a id="topic_rtd_cjh_15"></a>
@@ -151,7 +151,7 @@
To configure YARN as the global resource manager in a HAWQ cluster, add the following property configuration to your `hawq-site.xml` file:
-```
+``` xml
<property>
<name>hawq_global_rm_type</name>
<value>yarn</value>
@@ -164,7 +164,7 @@
If you set the global resource manager to YARN, you must also configure the following properties in `hawq-site.xml`:
-```
+``` xml
<property>
<name>hawq_rm_yarn_address</name>
<value>localhost:8032</value>
@@ -221,7 +221,7 @@
**Note:** When you use high availability in YARN, HAWQ ignores the values specified for `hawq_rm_yarn_address` and `hawq_rm_yarn_scheduler_address` in `hawq-site.xml` and uses the values specified in `yarn-client.xml` instead.
-```
+``` xml
<property>
<name>yarn.resourcemanager.ha</name>
<value>{0}:8032,{1}:8032</value>
diff --git a/resourcemgmt/best-practices.md b/resourcemgmt/best-practices.md
index 55f6aac..db69871 100644
--- a/resourcemgmt/best-practices.md
+++ b/resourcemgmt/best-practices.md
@@ -6,10 +6,10 @@
The following is a list of high-level best practices for optimal resource management:
-- Make sure segments do not have identical IP addresses. See [Segments Do Not Appear in gp\_segment\_configuration](/200/hawq/troubleshooting/Troubleshooting.html) for an explanation of this problem.
+- Make sure segments do not have identical IP addresses. See [Segments Do Not Appear in gp\_segment\_configuration](/20/troubleshooting/Troubleshooting.html) for an explanation of this problem.
- Configure all segments to have the same resource capacity. See [Configuring Segment Resource Capacity](ConfigureResourceManagement.html).
- To prevent resource fragmentation, ensure that your deployment's segment resource capacity \(standalone mode\) or YARN node resource capacity \(YARN mode\) is a multiple of all virtual segment resource quotas. See [Configuring Segment Resource Capacity](ConfigureResourceManagement.html) \(HAWQ standalone mode\) and [Setting HAWQ Segment Resource Capacity in YARN](YARNIntegration.html).
-- Ensure that enough registered segments are available and usable for query resource requests. If the number of unavailable or unregistered segments is higher than a set limit, then query resource requests are rejected. Also ensure that the variance of dispatched virtual segments across physical segments is not greater than the configured limit. See [Rejection of Query Resource Requests](/200/hawq/troubleshooting/Troubleshooting.html).
+- Ensure that enough registered segments are available and usable for query resource requests. If the number of unavailable or unregistered segments is higher than a set limit, then query resource requests are rejected. Also ensure that the variance of dispatched virtual segments across physical segments is not greater than the configured limit. See [Rejection of Query Resource Requests](/20/troubleshooting/Troubleshooting.html).
- Use multiple master and segment temporary directories on separate, large disks (2TB or greater) to load balance writes to temporary files (for example, `/disk1/tmp /disk2/tmp`). For a given query, HAWQ will use a separate temp directory (if available) for each virtual segment to store spill files. Multiple HAWQ sessions will also use separate temp directories where available to avoid disk contention. If you configure too few temp directories, or you place multiple temp directories on the same disk, you increase the risk of disk contention or running out of disk space when multiple virtual segments target the same disk.
- Configure minimum resource levels in YARN, and tune the timeout of when idle resources are returned to YARN. See [Tune HAWQ Resource Negotiations with YARN](YARNIntegration.html).
- Make sure that the property `yarn.scheduler.minimum-allocation-mb` in `yarn-site.xml` is an equal subdivision of 1GB. For example, 1024, 512. See [Setting HAWQ Segment Resource Capacity in YARN](YARNIntegration.html#topic_pzf_kqn_c5).
diff --git a/root-index.md.erb b/root-index.md.erb
deleted file mode 100644
index b23801d..0000000
--- a/root-index.md.erb
+++ /dev/null
@@ -1,34 +0,0 @@
----
-title: Pivotal HDB 2.0 Documentation
----
-
-<ol class="class-list">
- <li>
- <a class="subnav" href="../hdb20/index.html">Pivotal HDB 2.0 Documentation</a>
- </li>
- <li>
- <a class="subnav" href="../hdb20/releasenotes/HAWQ20ReleaseNotes.html">Release Notes</a>
- </li>
- <li>
- <a class="subnav" href="./requirements/system-requirements.html">System Requirements</a>
- </li>
- <li>
- <a class="subnav" href="./install/index.html">Installing Pivotal HDB</a>
- </li>
- <li>
- <a class="subnav" href="./overview/HAWQOverview.html">System Overview</a>
- </li>
- <li>
- <a class="subnav" href="./admin/RunningHAWQ.html">Running a HAWQ Cluster</a>
- </li>
- <li>
- <a class="subnav" href="./resourcemgmt/index.html">Managing Resources</a>
- </li>
- <li>
- <a class="subnav" href="./clientaccess/index.html">Managing Client Access</a>
- </li>
- <li>
- <a class="subnav" href="./ddl/ddl.html">Defining Database Objects</a>
- </li>
-
-</ol>
diff --git a/troubleshooting/Troubleshooting.html.md.erb b/troubleshooting/Troubleshooting.html.md.erb
new file mode 100644
index 0000000..034f956
--- /dev/null
+++ b/troubleshooting/Troubleshooting.html.md.erb
@@ -0,0 +1,101 @@
+---
+title: Troubleshooting
+---
+
+This chapter describes how to resolve common problems and errors that occur in a HAWQ system.
+
+
+
+## Query Performance Issues<a id="topic_dwd_rnx_15"></a>
+
+**Problem:** Query performance is slow.
+
+**Cause:** There can be multiple reasons why a query might be performing slowly. For example, the locality of data distribution, the number of virtual segments, or the number of hosts used to execute the query can all affect its performance. The following procedure describes how to investigate query performance issues.
+
+### How to Investigate Query Performance Issues<a id="task_ayl_pbw_c5"></a>
+
+A query is not executing as quickly as you would expect. Here is how to investigate possible causes of slowdown:
+
+1. Check the health of the cluster.
+ 1. Are any DataNodes, segments or nodes down?
+ 2. Are there many failed disks?
+
+2. Check table statistics. Have the tables involved in the query been analyzed?
+3. Check the plan of the query and run [`EXPLAIN ANALYZE`](/20/reference/sql/EXPLAIN.html) to determine the bottleneck.
+ Sometimes, there is not enough memory for some operators, such as Hash Join, or spill files are used. If an operator cannot perform all of its work in the memory allocated to it, it caches data on disk in *spill files*. Compared with no spill files, a query will run much slower.
+
+4. Check data locality statistics using [`EXPLAIN ANALYZE`](/20/reference/sql/EXPLAIN.html). Alternately you can check the logs. Data locality result for every query could also be found in the log of HAWQ. See [Data Locality Statistics](../query/query-performance.html#topic_amk_drc_d5) for information on the statistics.
+5. Check resource queue status. You can query view `pg_resqueue_status` to check if the target queue has already dispatched some resource to the queries, or if the target queue is lacking resources. See [Checking Existing Resource Queues](/20/resourcemgmt/ResourceQueues.html#topic_lqy_gls_zt).
+6. Analyze a dump of the resource manager's status to see more resource queue status. See [Analyzing Resource Manager Status](/20/resourcemgmt/ResourceQueues.html#topic_zrh_pkc_f5).
+
+## Rejection of Query Resource Requests<a id="topic_vm5_znx_15"></a>
+
+**Problem:** HAWQ resource manager is rejecting query resource allocation requests.
+
+**Cause:** The HAWQ resource manager will reject resource query allocation requests under the following conditions:
+
+- **Too many physical segments are unavailable.**
+
+ HAWQ resource manager expects that the physical segments listed in file `$GPHOME/etc/slaves` are already registered and can be queried from table `gp_segment_configuration`.
+
+ If the resource manager determines that the number of unregistered or unavailable HAWQ physical segments is greater than [hawq\_rm\_rejectrequest\_nseg\_limit](../reference/guc/parameter_definitions.html#hawq_rm_rejectrequest_nseg_limit), then the resource manager rejects query resource requests directly. The purpose of rejecting the query is to guarantee that queries are run in a full size cluster. This makes diagnosing query performance problems easier. The default value of `hawq_rm_rejectrequest_nseg_limit` is 0.25, which means that if more than 0.25 \* the number segments listed in `$GPHOME/etc/slaves` are found to be unavailable or unregistered, then the resource manager rejects the query's request for resources. For example, if there are 15 segments listed in the slaves file, the resource manager calculates that no more than 4 segments (0.25 \* 15) can be unavailable
+
+ In most cases, you do not need to modify this default value.
+
+- **There are unused physical segments with virtual segments allocated for the query.**
+
+ The limit defined in [hawq\_rm\_tolerate\_nseg\_limit](../reference/guc/parameter_definitions.html#hawq_rm_tolerate_nseg_limit) has been exceeded.
+
+- **Virtual segments have been dispatched too unevenly across physical segments.**
+
+ To ensure best query performance, HAWQ resource manager tries to allocate virtual segments for query execution as evenly as possible across physical segments. However, there can be variance in allocations. HAWQ will reject query resource allocation requests that have a variance greater than the value set in [hawq\_rm\_nvseg\_variance\_amon\_seg\_limit](../reference/guc/parameter_definitions.html#hawq_rm_nvseg_variance_amon_seg_limit)
+
+ For example, one query execution causes nine (9) virtual segments to be dispatched to two (2) physical segments. Assume that one segment has been allocated seven (7) virtual segments and another one has allocated two (2) virtual segments. Then the variance between the segments is five (5). If `hawq_rm_nvseg_variance_amon_seg_limit` is set to the default of one (1), then the allocation of resources for this query is rejected and will be reallocated later. However, if a physical segment has five virtual segments and the other physical segment has four (4), then this resource allocation is accepted.
+
+**Solution:** Check on the status of the nodes in the cluster. Restart existing nodes, if necessary, or add new nodes. Modify the [hawq\_rm\_nvseg\_variance\_amon\_seg\_limit](../reference/guc/parameter_definitions.html#hawq_rm_nvseg_variance_amon_seg_limit) (although note that this can affect query performance.)
+
+## Queries Cancelled Due to High VMEM Usage<a id="topic_qq4_rkl_wv"></a>
+
+**Problem:** Certain queries are cancelled due to high virtual memory usage. Example error message:
+
+``` pre
+ERROR: Canceling query because of high VMEM usage. Used: 1748MB, available 480MB, red zone: 9216MB (runaway_cleaner.c:135) (seg74 bcn-w3:5532 pid=33619) (dispatcher.c:1681)
+```
+
+**Cause:** This error occurs when the virtual memory usage on a segment exceeds the virtual memory threshold, which is can configured as a percentage through the [runaway\_detector\_activation\_percent](../reference/guc/parameter_definitions.html#runaway_detector_activation_percent).
+
+If the amount of virtual memory utilized by a physical segment exceeds the calculated threshold, then HAWQ begins terminating queries based on memory usage, starting with the query that is consuming the largest amount of memory. Queries are terminated until the percentage of utilized virtual memory is below the specified percentage.
+
+**Solution:** Try temporarily increasing the value of `hawq_re_memory_overcommit_max` to allow specific queries to run without error.
+
+Check `pg_log` files for more memory usage details on session and QE processes. HAWQ logs terminated query information such as memory allocation history and context information as well as query plan operator memory usage information. This information is sent to the master and segment instance log files.
+
+## Segments Do Not Appear in gp\_segment\_configuration<a id="topic_hlj_zxx_15"></a>
+
+**Problem:** Segments have successfully started, but cannot be found in table `gp_segment_configuration`.
+
+**Cause:** Your segments may have been assigned identical IP addresses.
+
+Some software and projects have virtualized network interfaces that use auto-configured IP addresses. This may cause some HAWQ segments to obtain identical IP addresses. The resource manager's fault tolerance service component will only recognize one of the segments with an identical IP address.
+
+**Solution:** Change your network's configuration to disallow identical IP addresses before starting up the HAWQ cluster.
+
+## Investigating Segments Marked As Down <a id="investigatedownsegment"></a>
+
+**Problem:** The [HAWQ fault tolerance service (FTS)](../admin/FaultTolerance.html) has marked a segment as down in the [gp_segment_configuration](../reference/catalog/gp_segment_configuration.html) catalog table.
+
+**Cause:** FTS marks a segment as down when a segment encounters a critical error. For example, a temporary directory on the segment fails due to a hardware error. Other causes might include network or communication errors, resource manager errors, or simply a heartbeat timeout. The segment reports critical failures to the HAWQ master through a heartbeat report.
+
+**Solution:** The actions required for recovering a segment varies depending upon the reason. In some cases, the segment is only marked as down temporarily until the heartbeat interval can recheck the segment's status. To investigate the reason why the segment was marked down, check the gp_configuration_history catalog table for a corresponding reason. See [Viewing the Current Status of a Segment](../admin/FaultTolerance.html#view_segment_status) for a description of various reasons that the fault tolerance service may mark a segment as down.
+
+## Handling Segment Resource Fragmentation<a id="topic_mdz_q2y_15"></a>
+
+Different HAWQ resource queues can have different virtual segment resource quotas, which can result in resource fragmentation. For example, a HAWQ cluster has 4GB memory available for a currently queued query, but the resource queues are configured to split four 512MB memory blocks in 4 different segments. It is impossible to allocate two 1GB memory virtual segments.
+
+In standalone mode, the segment resources are all exclusively occupied by HAWQ. Resource fragmentation can occur when segment capacity is not a multiple of a virtual segment resource quota. For example, a segment has 15GB memory capacity, but the virtual segment resource quota is set to 2GB. The maximum possible memory consumption in a segment is 14GB. Therefore, you should configure segment resource capacity as a multiple of all virtual segment resource quotas.
+
+In YARN mode, resources are allocated from the YARN resource manager. The HAWQ resource manager acquires a YARN container by 1 vcore. For example, if YARN reports that a segment having 64GB memory and 16 vcore is configured for YARN applications, HAWQ requests YARN containers by 4GB memory and 1 vcore. In this manner, HAWQ resource manager acquires YARN containers on demand. If the capacity of the YARN container is not a multiple of the virtual segment resource quota, resource fragmentation may occur. For example, if the YARN container resource capacity is 3GB memory 1 vcore, one segment may have 1 or 3 YARN containers for HAWQ query execution. In this situation, if the virtual segment resource quota is 2GB memory, then HAWQ will always have 1 GB memory that cannot be utilized. Therefore, it is recommended to configure YARN node resource capacity carefully to make YARN container resource quota as a multiple of all virtual segment resource quotas. In addition, make sure your CPU to memory ratio is a multiple of the amount configured for `yarn.scheduler.minimum-allocation-mb`. See [Setting HAWQ Segment Resource Capacity in YARN](/20/resourcemgmt/YARNIntegration.html#topic_pzf_kqn_c5) for more information.
+
+If resource fragmentation occurs, queued requests are not processed until either some running queries return resources or the global resource manager provides more resources. If you encounter resource fragmentation, you should double check the configured capacities of the resource queues for any errors. For example, an error might be that the global resource manager container's memory to core ratio is not a multiple of virtual segment resource quota.
+
+
