CaQL Usage
==========

CaQL, the Catalog Query Language, is a simplified SQL language and C
api for catalog operations.  This api unifies and supplants the
existing heap, index, systable, and syscache api's.  In addition, this
api can automatically use and enforce the correct locking and snapshot
modes for the specified catalog operation.  The CaQL conversion
project is the initial phase of the Unified Catalog Service, an
initiative which attempts to address many deficiencies in catalog
correctness, stability, extensibility, scalability, and performance.
While the basic api is simple, it has sufficient extensions to support
legacy requirements for unusual locking or index operations.  In this
document, the camelcase "CaQL" refers to the SQL-like language, while
the lowercase "caql" prefix describes functions in the catquery.c
module.

The C api supports a variety of methods that emulate the existing
interfaces, plus additional functionality.  These functions take two
common arguments: a cqContext and a cq_list.

A cqContext encapsulates all the scan, locking, and snapshot state for
a query.  If you supply a null cqContext, catquery will manufacture a
default context with the appropriate locking and snapshot modes.  caql
also has several functions to initialize the context to use special
locking and snapshot modes to support legacy requirements.

A cq_list is a CaQL statement plus a variable-length list of "Datum"
arguments, bound in a cql() macro declaration.  The cq_list must have
sufficient arguments to match the number of bind values (bindvals) in
the WHERE clause.

CaQL - the language
===================

CaQL is a small subset of SQL.  The following operations are supported:

SELECT * FROM <tablename> [WHERE <colname> = <bindval> [ AND ...]] [FOR UPDATE]
SELECT COUNT(*) ...
SELECT <colname> ...

DELETE FROM <tablename> [WHERE <colname> = <bindval> [ AND ...]] 

For a SELECT statement, the "FOR UPDATE" clause changes the default
locking from AccessShare to RowExclusive.  The SELECT list only
supports "*", "COUNT(*)", or a single column name (where the column is
of type Oid, Name, or Text).  The option WHERE clause supports an
ANDed list of "<colname> = <bindval>" expressions.  In the current
implementation, only five bind values (labeled ":1" to ":5") are
supported.

Extended Usage - sorting/ordering
=================================

Clients may use the optional "ORDER BY <colname> [, <colname...]"
clause to specify an ordering when iterating over a set of tuples.  If
no ordering is specified the tuple output order is random.  The ORDER
BY clause immediately precedes the FOR UPDATE (or terminates the
SELECT statement if FOR UPDATE is not specified).  CaQL also supports
GreaterThan/LessThan comparison in the WHERE clause for this case, eg
<,>,<=,>=.  In the current implementation, ordering requires the
existence of a matching index.  Note that SQL-style
ASCENDING/DESCENDING is not necessary: clients use caql_getnext() or
caql_getprev() to ascend or descend as they fetch tuples.

Sample Usage
============

    /* 
     * Example 1: find the first tuple for resource type and 
     * lock it for update 
     */

    HeapTuple   tuple;

    tuple = caql_getfirst(
            NULL,
            cql("SELECT * FROM pg_resourcetype"
                " WHERE resname = :1 FOR UPDATE",
                CStringGetDatum(pNameIn)));


    /* 
     * Example 2: build a context using an existing open relation, and see
     * if any queues match the supplied name 
     */
    pg_resqueue_rel = heap_open(ResQueueRelationId, RowExclusiveLock);
    if (caql_getcount(
            caql_addrel(cqclr(&cqc), pg_resqueue_rel),
            cql("SELECT COUNT(*) FROM pg_resqueue WHERE rsqname = :1",
                CStringGetDatum(stmt->queue))))
    {
        ereport(ERROR,
                (errcode(ERRCODE_DUPLICATE_OBJECT),
                 errmsg("resource queue \"%s\" already exists",
                        stmt->queue)));
    }


    /* Example 3: drop the extended attributes for this queue */
    int numDel;
    numDel =
        caql_getcount(
                NULL,
                cql("DELETE FROM pg_resqueuecapability WHERE resqueueid = :1",
                    ObjectIdGetDatum(queueid))
                ); 


    /* 
     * Example 4: build a context using an existing open relation, 
     * and iterate over all of the matching tuples 
     */
    gp_db_interface_rel = heap_open(GpDbInterfacesRelationId, AccessShareLock);

    pcqCtx = caql_beginscan(
            caql_addrel(cqclr(&cqc), gp_db_interface_rel),
            cql("SELECT * FROM gp_db_interfaces "
                " WHERE dbid = :1 ",
                ObjectIdGetDatum(c->dbid)));

    while (HeapTupleIsValid(tuple = caql_getnext(pcqCtx)))
    {
     ...
    }

    /* Finish up scan and close appendonly catalog. */
    caql_endscan(pcqCtx);
    heap_close(gp_db_interface_rel, AccessShareLock);


    /* 
     * Example 5: select only the dbid column from first matching tuple 
     * of this table, and set a count of how many tuples matched (0, 1, >1) 
     */

    Oid dbid;
    int fetchCount;

    dbid = caql_getoid_plus(
            NULL,
            &fetchCount,
            NULL,
            cql("SELECT dbid FROM gp_segment_configuration "
						" WHERE content = :1 "
						  " AND role = :2 ",
                Int16GetDatum(contentid),
						CharGetDatum('m')));


    /* 
     * Example 6: select only the constraint name from first matching tuple 
     * of this table
     */

	char	   *cname;

    cname = caql_getcstring(
             NULL,
          	   cql("SELECT conname FROM pg_constraint "
				   " WHERE oid = :1 ",
				  ObjectIdGetDatum(constraintId)));


Constructing and Modifying the caql context
===========================================

Almost all of the caql functions take a cqContext (caql context)
argument.  This context maintains information on table locking,
visibility, and fetch iteration status.  If a null context argument is
supplied, the caql function will internally generate a default
context.  While functions like caql_getfirst() and caql_getoid() only
have a context for the duration of their execution, The
caql_beginscan() function returns an initialized context which is a
required argument for subsequent caql_getnext() and caql_endscan()
functions.  

The most basic modification to the context is to add an existing,
open, locked relation.  If this relation is not supplied, the caql
function will open and lock the tables specified in the cql() query,
and close them at the end of execution (or at caql_endscan() for an
iterator).  If the relation is supplied using caql_addrel(), the caql
function will select tuples from the supplied relation, and leave it
open at the end of execution.  This pattern is common when multiple
SELECTs and UPDATEs are performed against the same table in a single
transaction.  The basic way to do this is:

    cqContext   cqc; /* declare the context on the stack */

    cqclr(&cqc); /* clear the context */

	caql_addrel(&cqc, some_open_relation); /* add an open relation */

However, since the context modification functions return a pointer to
the modified cqContext, you can nest or "chain" them:

    /* 
     * clear the context, add open relation pg_resqueue_rel, and supply it
     * to caql_getcount() 
     */
    numDel =
        caql_getcount(
            caql_addrel(cqclr(&cqc), pg_resqueue_rel),
            cql("DELETE FROM pg_resqueue WHERE rsqname = :1",
                CStringGetDatum(stmt->queue))))

Additional context modification function are caql_indexOK(),
caql_lockmode(), caql_snapshot(), and caql_syscache().  These
functions mainly exist to support legacy code requirements for the
initial CaQL conversion, enforcing particular patterns of index,
heapscan or syscache usage.  As the Unified Catalog Service evolves to
an external service, usage of these functions may be deprecated, as
the underlying implementation will be very different.  Best practice
is to avoid their usage if possible.

CaQL Context Lifetime
=====================

The caql context is valid from caql_beginscan() to caql_endscan().
After caql_endscan(), any open tables are closed, and the fetched
tuple is freed.  However, if a context pointer is supplied to
caql_getfirst() or caql_getfirst_only(), the tuple is copied out, so
the caql_cntext contains a valid pointer to the last fetched tuple
*after* the call.  In addition, if the relation is supplied using
caql_addrel(), then the caql context relation pointer remains valid
until heap_close() is called for the supplied relation.

INSERT, UPDATE, DELETE
======================

CaQL also supports INSERT, UPDATE, and DELETE operations in a
cursor-like context.  The caql context contains a pointer to the last
tuple fetched.  Several functions can operate upon the last, or
"current" tuple:

 void caql_delete_current(cqContext *pCtx);
 void caql_update_current(cqContext *pCtx, HeapTuple tup);
 HeapTuple caql_modify_current(cqContext *pCtx, Datum *replValues,
		   					 bool *replIsnull,
							 bool *doReplace);

caql_delete_current() performs a simple_heap_delete() on the current
tuple, and caql_update_current() performs a simple_heap_update() plus
CatalogUpdateIndexes().  caql_modify_current() peforms
heap_modify_tuple() on current.

CaQL supports an "INSERT" statement in caql_beginscan() which obtains
the correct table locking and maintains it until caql_endscan(), eg:

	/* open pg_proc_callback */
	pcqCtx = caql_beginscan(
			NULL,
			cql("INSERT INTO pg_proc_callback ",
				NULL));

	/* Build the tuple and insert it */
	nulls[Anum_pg_proc_callback_profnoid - 1]	  = false;
	nulls[Anum_pg_proc_callback_procallback - 1]  = false;
	nulls[Anum_pg_proc_callback_promethod - 1]	  = false;
	values[Anum_pg_proc_callback_profnoid - 1]	  = ObjectIdGetDatum(profnoid);
	values[Anum_pg_proc_callback_procallback - 1] = ObjectIdGetDatum(procbk);
	values[Anum_pg_proc_callback_promethod - 1]	  = CharGetDatum(promethod);

	tup = caql_form_tuple(pcqCtx, values, nulls);
	
	/* Insert tuple into the relation */
	caql_insert(pcqCtx, tup);  /* implicit update of index as well */

	caql_endscan(pcqCtx);

Note that the INSERT statement does not have a VALUES clause, or take
any arguments.  The actual insertion happens when a tuple is supplied
to caql_insert().  The caql_form_tuple() performs a heap_form_tuple
for the table specified by INSERT, and caql_insert() performs a
simple_heap_insert() and CatalogUpdateIndexes().  The caql_insert()
statement does not require the usage of the cql("INSERT INTO...")
statement -- it also works with caql contexts from 
cql("SELECT ... FOR UPDATE"), eg:

	pg_type_desc = heap_open(TypeRelationId, RowExclusiveLock);

	/* initialize a caql context for the subsequent INSERT and
	   UPDATE operations after the caql_getfirst()  
	   */
	pcqCtx = caql_addrel(cqclr(&cqc), pg_type_desc);

	tup = caql_getfirst(
			pcqCtx,
			cql("SELECT * FROM pg_type "
				" WHERE typname = :1 "
				" AND typnamespace = :2 "
				" FOR UPDATE ",
				CStringGetDatum((char *) typeName),
				ObjectIdGetDatum(typeNamespace)));

	if (HeapTupleIsValid(tup))
	{
		/*
		 * Okay to update existing shell type tuple
		 */
		tup = caql_modify_current(pcqCtx,
								  values,
								  nulls,
								  replaces);

		caql_update_current(pcqCtx, tup);
		/* and Update indexes (implicit) */
	}
	else
	{
		tup = caql_form_tuple(pcqCtx, values, nulls);
		/* Insert tuple into the relation */
		typeObjectId = caql_insert(pcqCtx, tup);
		/* and Update indexes (implicit) */
	}

	heap_close(TypeRelationId, RowExclusiveLock);

In this example, the caql context is "preloaded" with a valid, open,
relation, and a single tuple is fetched.  Since caql_getfirst() is
used, the context has a valid pointer to a copy of a tuple after the
call.  If the tuple is valid, it is updated, else a new tuple is
constructed and inserted into the open table.

Operational Details
===================

The calico.pl module extracts the CaQL statements from the cql()
macros in the C source files and generates a file called catquery.c.
This file contains a set of "basic" functions which perform the
underlying primitive database operations for the associated CaQL
statement.  calico.pl uses gperf, the gnu perfect hash utility, to
construct a static dispatch table that maps the CaQL statements to the
basic functions.  At runtime, the caql functions hash the cql()
statements to find and execute the associated basic database
functions.  All of the basic functions are pre-constructed during the
initial build and compilation of the database engine -- using a novel
CaQL statement at runtime results in an error.  The make system also
constructs caql.md5, the md5 hashes for the c source files used to
generate catquery.c.  The installcheck "caql" test compares the md5
signatures for the current set of sources with the historic signatures
for the catquery source files to ensure that the current postgres
executable is up-to-date.

Painful SysCache Brain Damage
=============================

I had to add some functions to handle SysCache routines that didn't
easily map into beginscan/getnext/endscan.

/* retrieve the last (current) tuple -- really only useful for 
 * caql_get_attname_scan()
 */
HeapTuple caql_get_current(cqContext *pCtx);

/*
 * adapted from original lsyscache.c/get_attnum()
 *
 *		Given the relation id and the attribute name,
 *		return the "attnum" field from the attribute relation.
 *
 *		Returns InvalidAttrNumber if the attr doesn't exist (or is dropped).
 */
AttrNumber caql_getattnumber(Oid relid, const char *attname);


/* ----------------------------------------------------------------
 * caql_getattname()
 *
 * The equivalent of SearchSysCacheCopyAttName - 
 * caql_getfirst(pCtx,	
 *    "SELECT * FROM pg_attribute
 *     WHERE attrelid = :relid
 *     AND attname = :attname
 *     AND attisdropped is false"
 *    );
 *
 * That is, find the existing ("undropped") attribute and return
 * a copy.
 * NOTE: need to be careful if this pCtx is used for update, as this
 * function "simulates" the ATTNAME cache lookup.
 * ----------------------------------------------------------------
 */
HeapTuple caql_getattname(cqContext *pCtx, Oid relid, const char *attname);

/* ----------------------------------------------------------------
 * caql_getattname_scan()
 *
 * The equivalent of SearchSysCacheAttName - 
 * caql_beginscan(pCtx,	
 *    "SELECT * FROM pg_attribute
 *     WHERE attrelid = :relid
 *     AND attname = :attname
 *     AND attisdropped is false"
 *    );
 *
 * That is, find the existing ("undropped") attribute and return
 * a context where the tuple is already fetched.  Retrieve the tuple
 * using caql_get_current()
 * NOTE: this is hideous.  My abject apologies.
 * NOTE: need to be careful if this pCtx is used for update, as this
 * function "simulates" the ATTNAME cache lookup.
 * ----------------------------------------------------------------
 */
cqContext *
caql_getattname_scan(cqContext *pCtx0, Oid relid, const char *attname)


/* catcache list-search interface.  Users of this must import catcache.h too */
/* 
 * In general, catquery will choose the syscache when the cql
 * statement contains an equality predicate on *all* of the syscache
 * primary key index columns, eg: 
 *
 *   cql("SELECT * FROM pg_amop WHERE amopopr = :1 and amopclaid = :2 ")
 *
 * will use the AMOPOPID syscache with index
 * AccessMethodOperatorIndexId.  However, the cql statement for a
 * list-search requires an equality predicate on a subset of the
 * initial columns of the index, with *all* of the index columns
 * specified in an ORDER BY clause, eg:
 *
 *   cql("SELECT * FROM pg_amop WHERE amopopr = :1 "
 *       " ORDER BY amopopr, amopclaid ")
 *
 * will use a syscache list-search if this cql statement is an
 * argument to caql_begin_CacheList().  However, the syscache will
 * *not* be used for this statement if it is supplied for
 * caql_beginscan(), since SearchSysCache() can only return (at most)
 * a single tuple.
 *
 * NOTE: caql_begin_CacheList() will assert (Insist!) at runtime if
 * the cql statement does not map to a syscache lookup.
 * NOTE: it may be possible to "collapse" this API into the existing
 * beginscan/getnext/endscan.
 */
struct catclist *caql_begin_CacheList(cqContext *pCtx, cq_list *pcql);
void caql_end_CacheList(struct catclist *cl)