#!/usr/bin/perl
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.
#
# $Header: //gpsql/feature/hd2.0/private-lili/src/include/catalog/tidycat.pl#2 $
#
# SLZY_HDR_END

use POSIX;
use Pod::Usage;
use Getopt::Long;
use Data::Dumper;
use strict;
use warnings;

# SLZY_POD_HDR_BEGIN
# WARNING: DO NOT MODIFY THE FOLLOWING POD DOCUMENT:
# Generated by sleazy.pl version 6 (release Mon Aug 20 12:30:03 2012)
# Make any changes under SLZY_TOP_BEGIN/SLZY_LONG_BEGIN

=head1 NAME

B<tidycat.pl> - generate catalog entries

=head1 VERSION

 This document describes version 34 of tidycat.pl, released
 Thu Oct 18 15:24:46 2012.

=head1 SYNOPSIS

B<tidycat.pl> 

Options:

    -help       brief help message
    -man        full documentation
    -dumpdef    output file for dump of serialized catalog data structures
    -dumpformat format options for dump file [perl, jason]
    -sqldef     output file for dump of catalog DDL statements
    -syscache   build syscache entries

=head1 OPTIONS

=over 8

=item B<-help>

    Print a brief help message and exits.

=item B<-man>

    Prints the manual page and exits.

=item B<-dumpdef> <filename>

        
Specify an optional filename to hold a dump of the serialized catalog
data structures.  The format of the dump file is determined by
dumpformat

=item B<-dumpformat> <filename>

    Specify a format for the dumpfile.  The only valid options are jason or perl.

=item B<-sqldef> <filename>

        
Specify an optional filename to hold the CREATE TABLE statements from
the tidycat definitions.  Note that these statements will contain the
tidycat WITH clause, which is not valid SQL.

=item B<-syscache>

    
If specified, rebuild syscache.h and syscache.c.  Note that this
option, like dumpdef, must read all catalog headers, ie in
src/include/catalog, the command:

   perl tidycat.pl -syscache *.h

constructs new versions of syscache.c and syscache.h.  

NOTE: Modification and extension of syscache entries is extremely rare.  
Usage of this option is discouraged.


=back

=head1 DESCRIPTION

tidycat.pl handles all of your stinky catalog problems, leaving a
fresh, clean scent.  Catalog tables require several sets of
co-ordinated modifications to multiple source files to define the
table and indexes, and (under some circumstances) the toast tables and
indexes (in toasting.h and toasting.c), as well as some special code
in catalog.c and bootparse.y to aid in bootstrap and upgrade.  tidycat
also updates a generated list of headers in pg_tidycat.h and the
catalog Makefile.  The original files are copied to a special
tidycat_backup directory in /tmp, and all generated files are written
to /tmp.  tidycat.pl uses a single definition statement to generate
the code associated with the table in multiple source files.  A sample
definition for the fictional pg_foobar.h follows:

  /* TIDYCAT_BEGINDEF

   CREATE TABLE pg_foobar
   with (camelcase=FooBar, shared=true, oid=true, relid=9991, reltype_oid=9992)
   (
   fooname name, -- name of foo bar
   foolimit real, -- max active count limit
   fooignore boolean, -- ignore foo in baz context
   );

   create unique index on pg_foobar(oid) with (indexid=9993);
   create index on pg_foobar(fooname) with (indexid=9994);

   TIDYCAT_ENDDEF
  */

The definition must begin and end with the
TIDYCAT_BEGINDEF/TIDYCAT_ENDDEF exactly as shown.  The CREATE TABLE
statement is almost identical to standard SQL, with the addition of a
special WITH clause for implementation-specific features of the
catalog entry.  Currently, the relid and reltype_oid must be specified
using unassigned oids from the unused_oids script.  The options are:

=over 8

=item CamelCase: (optional)

If your tablename is a compound name, the index definitions look a
little nicer if you define an appropriate camelcase name.  Otherwise,
the default version of the name is the tablename, minus the "pg_" prefix, 
initial letter capitalized.

=item shared: (false by default)

Whether the table is local to each database or shared by all.

=item oid: (true by default)

Whether the table has an auto-generated oid column.

=item relid: (required)

The relid of the table in pg_class.  Use unused_oids to find one.

=item reltype_oid: (required for all post-3.3 tables)

The static reltype in pg_type (necessary for upgrade).  Use
unused_oids to find one.

=item toast_oid: (required for all tables with text or array columns)

The oid of the toast table (see toasting.h).  Use unused_oids to find
one.  tidycat will automatically detect if the table definition
requires a toast table and return an error if it is not specified.

=item toast_index: (required for all tables with toast_oid)

The oid of the index of the toast table (see toasting.h).  Use
unused_oids to find one.

=item toast_reltype: (required for all toast tables post-3.3)

The static reltype of the toast table in pg_type (necessary for
upgrade). Use unused_oids to find one.

=item content: (optional)

The "content" is only for catalog tables with non-standard content
management.  "Normal" catalog tables are replicated from the master to
all the segments.  Non-standard tables fall into three categories:
MASTER_ONLY, SEGMENT_LOCAL, and PERSISTENT. Don't add any new
non-standard tables.  Please.  Note that this flag controls the
generation of validation logic for checkcat; it does not control the
catalog table tuple replication mechanisms.

=back

Similarly, index definitions are unique or non-unique, and require an
indexid (and an optional indexname).

Running tidycat.pl against pg_foobar.h adds the following section
after the definition:

  /* TIDYCAT_BEGIN_CODEGEN 
  
     WARNING: DO NOT MODIFY THE FOLLOWING SECTION: 
     Generated by tidycat.pl version 3.
     on Tue Dec  8 12:50:21 2009
  */
  
  
  /*
   TidyCat Comments for pg_foobar:
    Table is shared, so catalog.c:IsSharedRelation is updated.
    Table has an Oid column.
    Table has static type (see pg_types.h).
  
  */
  
  /* ----------------
   *		pg_foobar definition.  cpp turns this into
   *		typedef struct FormData_pg_foobar
   * ----------------
   */
  #define FooBarRelationId	9991
  
  CATALOG(pg_foobar,9991) BKI_SHARED_RELATION
  {
  	NameData	fooname;	/* name of foo bar */
  	float4		foolimit;	/* max active count limit */
  	bool		fooignore;	/* ignore foo in baz context */
  } FormData_pg_foobar;
  
  
  /* ----------------
   *		Form_pg_foobar corresponds to a pointer to a tuple with
   *		the format of pg_foobar relation.
   * ----------------
   */
  typedef FormData_pg_foobar *Form_pg_foobar;
  
  
  /* ----------------
   *		compiler constants for pg_foobar
   * ----------------
   */
  #define Natts_pg_foobar				3
  #define Anum_pg_foobar_fooname		1
  #define Anum_pg_foobar_foolimit		2
  #define Anum_pg_foobar_fooignore	3
  
  
  /* TIDYCAT_END_CODEGEN */
  

The generated code contains a CATALOG macro/struct definition for the
table, where the SQL datatypes are converted to C types.  The naming
and comments follow established conventions.

Additional modifications are made to indexing.h:

  /* relation id: 9991 - pg_foobar 20091208 */
  DECLARE_UNIQUE_INDEX(pg_foobar_oid_index, 9993, on pg_foobar using btree(oid oid_ops));
  #define FooBarOidIndexId 9993

  /* relation id: 9991 - pg_foobar 20091208 */
  DECLARE_INDEX(pg_foobar_fooname_index, 9994, on pg_foobar using btree(fooname name_ops));
  #define FooBarFoonameIndexId 9994


And the function IsSharedRelation() in catalog.c:
  bool 
  IsSharedRelation(Oid relationId)
  {
      /* These are the shared catalogs (look for BKI_SHARED_RELATION) */
      if (relationId == AuthIdRelationId ||
	(...much code...)
  /* relation id: 9991 - pg_foobar 20100105 */
  relationId == FooBarRelationId ||

Note that IsSharedRelation is only updated for shared tables.

And Boot_CreateStmt in bootparse.y:

  Boot_CreateStmt:
	  	  XCREATE optbootstrap optsharedrelation optwithoutoids boot_ident oidspec LPAREN
				{
	(...much code...)
  /* relation id: 9991 - pg_foobar 20100105 */
                            case FooBarRelationId:
                                typid = PG_FOOBAR_RELTYPE_OID;
                                break;

And pg_type.h:

  /* relation id: 9991 - pg_foobar 20100105 */
  DATA(insert OID = 9992 ( pg_foobar      PGNSP PGUID -1 f c t \054 9991 0 
	   record_in record_out record_recv record_send - d x f 0 -1 0 
       _null_ _null_ ));
  #define PG_FOOBAR_RELTYPE_OID 9992


=head2 JSON document

In src/include/catalog, the command:

   perl tidycat.pl -dd foo.json -df json *.h

will generate a JSON document describing all of the catalog tables.
This file is installed under tools/bin/gppylib/data, and gpcheckcat
uses this data to generate check queries for foreign key constraint.

=head1 CAVEATS

tidycat does not modify the original files -- it writes modified
versions of the files to /tmp.  You need to copy over the originals
with the generated files manually.  If you need to restore the
originals you can use the copies from the tidycat_backup directory.
Multiple cycles of tidycat with changing definitions can leave junk in
/tmp, and you might copy that junk into you source tree.  Do not copy
over any generated files that are older than your latest backup
directory.


=head1 AUTHORS

Apache HAWQ

Address bug reports and comments to: dev@hawq.apache.org

=cut
# SLZY_POD_HDR_END


my $glob_id = "";

my $glob_platform;
my $glob_tabwidth = 4;
my $glob_faketab = 0;
my $glob_tabstr = "\t";
#my $glob_tabstr = " " x $glob_tabwidth; 
my $glob_tmpdir = "/tmp";

# SLZY_GLOB_BEGIN
my $glob_glob;
# SLZY_GLOB_END

sub glob_validate
{

	if ($glob_glob->{dumpdef} && 
		defined($glob_glob->{dumpformat}) &&
		length($glob_glob->{dumpformat}))
	{
		die ("bad dump format: $glob_glob->{dumpformat}")
			unless ($glob_glob->{dumpformat} =~ m/jason|json|perl/i);
	}
	else
	{
		$glob_glob->{dumpformat} = "perl";
	}


#    print "loading...\n" ;
}

# SLZY_CMDLINE_BEGIN
# WARNING: DO NOT MODIFY THE FOLLOWING SECTION:
# Generated by sleazy.pl version 6 (release Mon Aug 20 12:30:03 2012)
# Make any changes under SLZY_TOP_BEGIN/SLZY_LONG_BEGIN
# Any additional validation logic belongs in glob_validate()

BEGIN {
	    my $s_help          = 0;     # brief help message
	    my $s_man           = 0;     # full documentation
	    my $s_dumpdef;               # output file for dump of serialized catalog data structures
	    my $s_dumpformat;            # format options for dump file [perl, jason]
	    my $s_sqldef;                # output file for dump of catalog DDL statements
	    my $s_syscache      = 0;     # build syscache entries

	my $slzy_argv_str;
	$slzy_argv_str = quotemeta(join(" ", @ARGV))
		if (scalar(@ARGV));

    GetOptions(
		'help|?'                     =>     \$s_help,
		'man'                        =>     \$s_man,
		'dumpdef|dd:s'               =>     \$s_dumpdef,
		'dumpformat|df|dumpfmt:s'    =>     \$s_dumpformat,
		'sqldef:s'                   =>     \$s_sqldef,
		'syscache'                   =>     \$s_syscache,
               )
        or pod2usage(2);

	pod2usage(-msg => $glob_id, -exitstatus => 1) if $s_help;
	pod2usage(-msg => $glob_id, -exitstatus => 0, -verbose => 2) if $s_man;
	
	
	$glob_glob = {};
	
	
	# version and properties from json definition
	$glob_glob->{_sleazy_properties} = {};
	$glob_glob->{_sleazy_properties}->{version} = '34';
	$glob_glob->{_sleazy_properties}->{COPYDATES} = '2009-2012';
	$glob_glob->{_sleazy_properties}->{slzy_date} = '1350599086';
	$glob_glob->{_sleazy_properties}->{slzy_argv_str} = $slzy_argv_str;
	
	$glob_glob->{dumpdef}        =  $s_dumpdef      if (defined($s_dumpdef));
	$glob_glob->{dumpformat}     =  $s_dumpformat   if (defined($s_dumpformat));
	$glob_glob->{sqldef}         =  $s_sqldef       if (defined($s_sqldef));
	$glob_glob->{syscache}       =  $s_syscache     if (defined($s_syscache));
	
	glob_validate();


}
# SLZY_CMDLINE_END


# DO NOT extend this list!  To ensure smooth upgrade, all new tables
# must have a static reltype_oid
my %dynamic_reltype_h = 
	(
 5000 => "gp_configuration",
 5002 => "gp_distribution_policy",
 5008 => "gp_master_mirroring",
 5003 => "gp_version_at_initdb",
 2600 => "pg_aggregate",
 2601 => "pg_am",
 2602 => "pg_amop",
 2603 => "pg_amproc",
 6105 => "pg_appendonly",
 2604 => "pg_attrdef",
 1249 => "pg_attribute", # special bootstrap case
 1261 => "pg_auth_members",
 1260 => "pg_authid",
 1248 => "pg_autovacuum",
 2605 => "pg_cast",
 1259 => "pg_class", # special bootstrap case
 2606 => "pg_constraint",
 2607 => "pg_conversion",
 1262 => "pg_database",
 2608 => "pg_depend",
 2609 => "pg_description",
 6040 => "pg_exttable",
 2610 => "pg_index",
 2611 => "pg_inherits",
 2612 => "pg_language",
 2613 => "pg_largeobject",
 2614 => "pg_listener",
 2615 => "pg_namespace",
 2616 => "pg_opclass",
 2617 => "pg_operator",
 5010 => "pg_partition",
 5011 => "pg_partition_rule",
 1136 => "pg_pltemplate",
 1255 => "pg_proc", # special bootstrap case
 2618 => "pg_rewrite",
 1214 => "pg_shdepend",
 2396 => "pg_shdescription",
 2619 => "pg_statistic",
 1213 => "pg_tablespace",
 2836 => "pg_toast_1255",
 2842 => "pg_toast_1260",
 2844 => "pg_toast_1262",
 2846 => "pg_toast_2396",
 2830 => "pg_toast_2604",
 2832 => "pg_toast_2606",
 2834 => "pg_toast_2609",
 2838 => "pg_toast_2618",
 2840 => "pg_toast_2619",
 2620 => "pg_trigger",
 1247 => "pg_type", # special bootstrap case
 5004 => "pg_window"
	);


# DO NOT extend this list!  To ensure smooth upgrade, all new tables
# with text or array columns must have toast tables and indexes
my %toast_tab_exception_h = 
	(
	 "gp_configuration_history" => 1,
	 "gp_configuration" => 1,
	 "gp_distribution_policy" => 1,
	 "gp_master_mirroring" => 1,
	 "gp_persistent_filespace_node" => 1,
	 "gp_san_configuration" => 1,
	 "gp_version_at_initdb" => 1,
	 "pg_aggregate" => 1,
	 "pg_appendonly" => 1,
	 "pg_appendonly_alter_column" => 1,
	 "pg_class" => 1,
	 "pg_exttable" => 1,
	 "pg_foreign_data_wrapper" => 1,
	 "pg_foreign_server" => 1,
	 "pg_foreign_table" => 1,
	 "pg_index" => 1,
	 "pg_partition_rule" => 1,
	 "pg_pltemplate" => 1,
	 "pg_stat_last_operation" => 1,
	 "pg_stat_last_shoperation" => 1,
	 "pg_tablespace" => 1,
	 "pg_type" => 1,
	 "pg_user_mapping" => 1
	);

# DO NOT extend this list!  To ensure smooth upgrade, all new toast
# tables must have a fixed reltype in pg_type.h
my %toast_reltype_exception_h = 
	(
	 "pg_attrdef" => 1,
	 "pg_constraint" => 1,
	 "pg_description" => 1,
	 "pg_proc" => 1,
	 "pg_rewrite" => 1,
	 "pg_statistic"=> 1,
	 "pg_authid" => 1,
	 "pg_database" => 1,
	 "pg_shdescription" => 1
	);

# DO NOT extend this list!  All new tidycat files should get
# registered in pg_tidycat.h
my %allfiles_exception_h = 
	(
	 "pg_aggregate.h" => 1,
	 "pg_am.h" => 1,
	 "pg_amop.h" => 1,
	 "pg_amproc.h" => 1,
	 "pg_aoseg.h" => 1,
	 "pg_appendonly.h" => 1,
	 "pg_appendonly_alter_column.h" => 1,
	 "pg_attrdef.h" => 1,
	 "pg_attribute.h" => 1,
	 "pg_auth_members.h" => 1,
	 "pg_authid.h" => 1,
	 "pg_autovacuum.h" => 1,
	 "pg_cast.h" => 1,
	 "pg_class.h" => 1,
	 "pg_constraint.h" => 1,
	 "pg_conversion.h" => 1,
	 "pg_database.h" => 1,
	 "pg_depend.h" => 1,
	 "pg_description.h" => 1,
	 "pg_extprotocol.h" => 1,
	 "pg_exttable.h" => 1,
	 "pg_filespace.h" => 1,
	 "pg_filespace_entry.h" => 1,
	 "pg_index.h" => 1,
	 "pg_inherits.h" => 1,
	 "pg_language.h" => 1,
	 "pg_largeobject.h" => 1,
	 "pg_listener.h" => 1,
	 "pg_namespace.h" => 1,
	 "pg_opclass.h" => 1,
	 "pg_operator.h" => 1,
	 "pg_partition.h" => 1,
	 "pg_partition_rule.h" => 1,
	 "pg_pltemplate.h" => 1,
	 "pg_proc.h" => 1,
	 "pg_resqueue.h" => 1,
	 "pg_rewrite.h" => 1,
	 "pg_shdepend.h" => 1,
	 "pg_shdescription.h" => 1,
	 "pg_statistic.h" => 1,
	 "pg_tablespace.h" => 1,
	 "pg_trigger.h" => 1,
	 "pg_type.h" => 1,
	 "pg_user_mapping.h" => 1,
	 "pg_window.h" => 1
 	);


sub getcomment1
{
	my $tname = shift;

	my $bigstr = <<"EOF_bigstr";
/* ----------------
 *TWOTABSTABLENAME definition.  cpp turns this into
 *TWOTABStypedef struct FormData_TABLENAME
 * ----------------
 */
EOF_bigstr

my $twotabs = $glob_tabstr x 2;

$bigstr =~ s/TWOTABS/$twotabs/gm;
$bigstr =~ s/TABLENAME/$tname/gm;

return $bigstr;

}

sub getcomment2
{
	my ($tname, $tform) = @_;

	my $bigstr = <<"EOF_bigstr";
/* ----------------
 *TWOTABSTFORM corresponds to a pointer to a tuple with
 *TWOTABSthe format of TABLENAME relation.
 * ----------------
 */
EOF_bigstr

my $twotabs = $glob_tabstr x 2;

$bigstr =~ s/TWOTABS/$twotabs/gm;
$bigstr =~ s/TABLENAME/$tname/gm;
$bigstr =~ s/TFORM/$tform/gm;

return $bigstr;

}
sub getcomment3
{
	my $tname = shift;

	my $bigstr = <<"EOF_bigstr";
/* ----------------
 *TWOTABScompiler constants for TABLENAME
 * ----------------
 */
EOF_bigstr

my $twotabs = $glob_tabstr x 2;

$bigstr =~ s/TWOTABS/$twotabs/gm;
$bigstr =~ s/TABLENAME/$tname/gm;

return $bigstr;

}

sub sqltype_to_ctype
{
	my $coltype = shift;
	my $ctype;

#	print $coltype, "\n";

	# list of valid C types for SQL types.  
	#
	# NOTE: not all sql types are valid for catalog tables, 
	# so do *NOT* extend this list unless you know what you are doing!!
	my %sql2ch =
		(
		 aclitem				  => "aclitem",
		 bigint					  => "bigint",
		 bool					  => "bool", # boolean is the real sqltype
		 boolean				  => "bool", 
		 bytea					  => "bytea",

		 # Note: a quoted_char (or "char") is a single C char
		 quoted_char			  => "char",

		 gpxlogloc				  => "gpxlogloc",
		 int2vector				  => "int2vector",
		 integer				  => "int4",
		 name					  => "NameData",
		 oid					  => "Oid",
		 oidvector				  => "oidvector",
		 real					  => "float4",
		 regproc				  => "regproc",
		 smallint				  => "int2",
		 text					  => "text",
		 tid					  => "tid",

		 # NOTE: both time and timestamp use a sleazy hack due to
		 # bootstrap nonsense
		 time					  => "time",
		 timestamp_with_time_zone => "timestamptz",

		 xid					  => "xid",
		);

	# "[]" for optional "array of" suffix

	my $isarray = ($coltype =~ m/\[\]$/);

	$coltype =~ s/\[\]$//;

	die "no C conversion for type: $coltype"
		unless (exists($sql2ch{$coltype}));

	$ctype = $sql2ch{$coltype};

	# make an array of 1
	$ctype .= "[1]" if ($isarray);

	return $ctype;
}

sub ctype_to_btree_op_prefix
{
	my $ctype = shift;

	return $ctype
		unless ($ctype =~ m/Oid|NameData|regproc/);

	$ctype = "name" if ($ctype =~ m/NameData/);
	$ctype = "oid" if ($ctype =~ m/Oid/);

	# NOTE: regproc is an oid typedef
	$ctype = "oid" if ($ctype =~ m/regproc/);

	return $ctype;
}

sub parseindex
{
	my ($alltabs, $bigstr) = @_;
	my $bUnique = 0;

	$bUnique = ($bigstr =~ m/create\s+unique\s+index/i);

	my $idef = {unique => $bUnique, with => {} };

	$bigstr =~ s/^\s*create\s+(unique\s+)?index\s+on//i;

#	print "f:",$bigstr,":f\n";

#	my @foo = ($bigstr =~ m/^s*(\w+)\s*\((.*)\)\s*(with\s*\(.*\))?/im);
#	my @foo = ($bigstr =~ m/\s*(\w+)\s*\(((?<!\)).*\))/i);
#   tablename (col1 (, col2)) 
	my @foo = ($bigstr =~ m/\s*(\w+)\s*\((\w+(\s*\,\s*\w+)*)\)/i);

#	print Data::Dumper->Dump(\@foo), "\n";

	die "bad index def: $bigstr" unless (2 < scalar(@foo));

	my $tname = shift @foo;
	my $colnamlist = shift @foo;
	my @cols = split(",", $colnamlist);

	my $icols = [];

	$tname = lc($tname);
	$tname =~ s/^\s+//;
	$tname =~ s/\s+$//;

	die "bad index def - no such table: $tname" unless (exists($alltabs->{$tname}));

	for my $c1 (@cols)
	{
		$c1 = lc($c1);
		$c1 =~ s/^\s+//;
		$c1 =~ s/\s+$//;

		die "bad index for $tname - no such col: $c1" 
			unless (exists($alltabs->{$tname}->{colh}->{$c1}));

		my $colop = $alltabs->{$tname}->{colh}->{$c1};

		$colop = ctype_to_btree_op_prefix($colop);
		$colop = $colop . "_ops";

		push @{$icols}, [$c1, $colop];
	}

	@foo = split(/\)/, $bigstr, 2);

	my $with = pop @foo;

#	print "WITH: $with \n";

	my @baz = ($with =~ m/^\s*(with\s*\(.*\))/is);

	die "bad index for $tname - no index oid: $bigstr" unless (scalar(@baz));

	my $indwithclause = shift @baz;

	$indwithclause =~ s/^\s*with\s*\(//is;
	$indwithclause =~ s/\)\s*$//s;

	@baz = split(",", $indwithclause);

	for my $withdef (@baz)
	{
		my @bzz = split("=", $withdef, 2);

		die "bad index with def for $tname: $withdef" unless (2 == scalar(@bzz));

		my $kk = shift @bzz;
		my $vv = shift @bzz;

		$kk =~ s/^\s+//;
		$kk =~ s/\s+$//;
		$kk = lc($kk);

		$vv =~ s/^\s+//;
		$vv =~ s/\s+$//;

		$idef->{with}->{$kk} = $vv;
	}

	$alltabs->{$tname}->{indexes} = []
		unless (exists(	$alltabs->{$tname}->{indexes}));
	
	$idef->{cols} = $icols;

	die "bad index def for $tname - no index oid: $bigstr" unless (exists($idef->{with}->{indexid}));

	$idef->{indexid} = $idef->{with}->{indexid};
	if (exists($idef->{with}->{indexname}))
	{
		$idef->{indexname} = $idef->{with}->{indexname};
	}

	push @{$alltabs->{$tname}->{indexes}}, $idef;
} # end parseindex

# parse foreign key definitions
sub parsefk
{
	my ($alltabs, $bigstr, $filnam) = @_;

	my @baz = ($bigstr =~ m/^\s*alter\s+table\s+(\w+)\s+add\s+(vector\_)?fk/i);

	my $tname = shift @baz;

	die "$filnam: bad foreign key - no such table: $tname" unless (exists($alltabs->{$tname}));

	# special case of vector fk for oidvector, oid array
	my $isvector = ($bigstr =~ m/add\s+vector\_fk/);

	$bigstr =~ s/^\s*alter\s+table\s+\w+\s+add\s+(vector\_)?fk//i;

	my @foo;

	# allow composite key, eg "add fk (k1[,k2...]) on ..."
	if ($bigstr =~ 
		m/\s*\(\s*(\w+(?:\s*,\s*\w+)*)\s*\)\s*on\s+(\w+)\s*\(\s*(\w+(?:\s*\,\s*\w+)*\s*)\)/i)
	{
		@foo = 
			($bigstr =~ 
			 m/\s*\(\s*(\w+(?:\s*,\s*\w+)*)\s*\)\s*on\s+(\w+)\s*\(\s*(\w+(?:\s*\,\s*\w+)*)\s*\)/i);
	}
	else # single key, no parents, ie "add fk k1 on ..."
	{
		@foo = 
			($bigstr =~ m/\s*(\w+)\s+on\s+(\w+)\s*\(\s*(\w+(?:\s*\,\s*\w+)*)\s*\)/i);
	}
#	print Data::Dumper->Dump(\@foo), "\n";

	die "$filnam: bad foreign key for table $tname: $bigstr" unless (2 < scalar(@foo));

	$alltabs->{$tname}->{foreign_keys} = []
		unless (exists($alltabs->{$tname}->{foreign_keys}));

	# NOTE: fk_list format supercededs original "foreign_keys" array...
	$alltabs->{$tname}->{fk_list} = []
		unless (exists($alltabs->{$tname}->{fk_list}));

	my $fknamlist  = shift @foo; # the foreign key column list
	my $pktname	   = shift @foo; # the primary key table name
	my $colnamlist = shift @foo; # the primary key cols

	# remove leading/trailing spaces around comma as well
	my @cols   = split(/\s*,\s*/, $colnamlist);
	my @fkcols = split(/\s*,\s*/, $fknamlist);
	
	for my $fkname (@fkcols)
	{
		die "$filnam: bad foreign key for table $tname - no such column: $fkname" 
			unless (exists($alltabs->{$tname}->{colh}->{$fkname}));
	}

	# XXX XXX: allow a "vector" fk for oidvector or oid array
	if ($isvector)
	{
		die "$filnam: bad vector foreign key for table $tname - too many columns"
			unless (1 == scalar(@fkcols));

		my $fkname = $fkcols[0];

		die "$filnam: bad vector foreign key $fkname for table $tname - must be an Oid vector or array: " 
			unless ($alltabs->{$tname}->{colh}->{$fkname} =~ 
					m/^(Oid\[1\]|oidvector)$/);

	}

	# old-style foreign key def only for regular, "scalar" keys
	push @{$alltabs->{$tname}->{foreign_keys}}, [\@fkcols, $pktname, \@cols]
		unless ($isvector);

	my $fkh = {type => "scalar", 
			   fkcols => \@fkcols, pktable => $pktname, pkcols => \@cols
	};

	$fkh->{type} = "vector" if ($isvector);

	push @{$alltabs->{$tname}->{fk_list}}, $fkh;
} # end parsefk

sub parsecols
{
	my ($alltabs, $tname, $bigstr) = @_;

#	print $bigstr;

	$bigstr =~ s/^\s*\(//s;
	$bigstr =~ s/\)\s*$//s;

#	print $bigstr;

	my @foo = split(/\n/, $bigstr);

	my $collist = [];
	my $ii = 0;

	my $tzcolname;
	my $tmcolname;

	my $precomment;

	my $colh = {}; # column data hash

	for my $lin (@foo)
	{
#		print $lin,"\n";

		unless (length($lin))
		{
			$precomment = ""
				unless (defined($precomment));

			# mark the blank lines (remove this later)
			$precomment .= "\n**TK_BLANK_LINE**";
			next;
		}

		if (($lin =~ m/^\s*\-\-/) || ($lin =~ m/^\s+$/))
		{
			$precomment = ""
				unless (defined($precomment));

			chomp($lin);
			$precomment .= "\n" . $lin;
		}
		else
		{
			# (optional quoted) word space word (or word space "char")
			die "bad col: $lin"
				unless ($lin =~ m/(\s*(\")*\w+(\")*\s+\w+)|\s*\w+\s+\"char\"/);

			# Note: timestamp fix - make into a single token
			$lin =~ 
				s/timestamp\s+with\s+time\s+zone/timestamp_with_time_zone/igm;

			# Note: quoted char (ie "char") substitution --
			# distinguish unquoted sql char, which is "character(1)",
			# or sql type bpchar, from postgresql-specific 
			# quoted char ("char"), which is a single C char
			$lin =~ 
				s/\"char\"/quoted_char/igm;

			# in sql, have a 
			# colname (with optional quotes) space coltype (followed
			# by optional array brackets), optionally followed by a
			# comma and/or a sql comment
			my @baz = 
				($lin =~ m/\s*((?:\")*\w+(?:\")*)\s+(\w+(?:\[\])?)\s*(\,)?\s*(\-\-.*)?/);

#			print Data::Dumper->Dump(\@baz), "\n";

			my $colname = shift @baz ;
			my $coltype = lc (shift @baz );

			# downcase colname unless it is quoted
			if ($colname !~ m/^\".*\"$/)
			{
				$colname = lc ($colname);
			}
			else
			{
				# remove quotes
				$colname =~ s/^\"(.*)\"$/$1/;
			}

			die ("duplicate colname: $colname for $tname")
				if (exists($colh->{$colname}));

			shift @baz ; # comma or undef

			my $postcomment; # the comment trailing the definition

			$postcomment = $baz[0] if (scalar(@baz) && defined($baz[0]));
			
			my $coldef = {colname => $colname, sqltype => $coltype};

			my $ctype = sqltype_to_ctype($coltype);
			$coldef->{ctype} = $ctype;

			$colh->{$colname} = $ctype;

			# track any timestamp columns specially
			if ($ctype =~ m/timestamp/)
			{
				if (defined($tzcolname))
				{
					# print '"colname" et al' if more than one
					# timestamp col
					$tzcolname .= " et al"
						unless ($tzcolname =~ m/et al$/);
				}
				else
				{
					$tzcolname = '"' . $colname . '"';
				}
			}
			# track any TimeADT columns specially
			if ($ctype =~ m/time$/)
			{
				if (defined($tmcolname))
				{
					# print '"colname" et al' if more than one
					# timestamp col
					$tmcolname .= " et al"
						unless ($tmcolname =~ m/et al$/);
				}
				else
				{
					$tmcolname = '"' . $colname . '"';
				}
			}

			$coldef->{postcomment} = $postcomment if (defined($postcomment));
			$coldef->{precomment} = $precomment if (defined($precomment));

			push @{$collist}, $coldef;

			$ii++;
			undef $precomment;
		}

	} # end for my lin
	
	# add the Oid column
	if ($alltabs->{$tname}->{with}->{oid})
	{
		die "name conflict: cannot have named oid column"
			if (exists($colh->{oid}));

		$colh->{oid} = "Oid";
	}

	$alltabs->{$tname}->{cols} = $collist;
	$alltabs->{$tname}->{colh} = $colh;

	# hack to track timestamp
	$alltabs->{$tname}->{tzhack} = $tzcolname
		if (defined($tzcolname));
	$alltabs->{$tname}->{tmhack} = $tmcolname
		if (defined($tmcolname));

#	print Data::Dumper->Dump(\@foo), "\n";

} # end parsecols

sub parsetab
{
	my ($alltabs, $bigstr, $filnam) = @_;
	
	my $tabdef = $bigstr;

	$bigstr =~ s/create\s+table//is;
	$bigstr =~ s/^\s+//s;
	$bigstr =~ s/\s+$//s;

	my @foo = ($bigstr =~ m/^\s*(\w+)/s);

	die "no tablename: $tabdef" unless (scalar(@foo));

	my $tname = shift @foo;

	$bigstr =~ s/$tname//s;
	$bigstr =~ s/^\s+//s;

	$tname =~ s/^\s+//;
	$tname =~ s/\s+$//;
	$tname = lc($tname);

	$alltabs->{$tname} = {tabdef_text => $tabdef, filename => $filnam };

	if ($bigstr =~ m/\s*with\s*\(.*\)/is)
	{
		# match a "WITH (...)" followed by "(" 
		@foo = ($bigstr =~ m/^\s*(with\s*\(.*\)(?=(\s*\()))/is);

		die "bad with: $bigstr" unless (scalar(@foo));

		my $withclause  = shift @foo;

		$alltabs->{$tname}->{with} = {text => $withclause};

		$bigstr =~ s/\s*(with\s*\(.*\)(?=(\s*\()))//is;
		$bigstr =~ s/^\s+//s;

		$withclause =~ s/^\s*with\s*\(//is;
		$withclause =~ s/\)\s*$//s;

		my @foo = split(",", $withclause);

		for my $withdef (@foo)
		{
			my @baz = split("=", $withdef, 2);

			die "bad with def: $withdef" unless (2 == scalar(@baz));

			my $kk = shift @baz;
			my $vv = shift @baz;

			$kk =~ s/^\s+//;
			$kk =~ s/\s+$//;
			$kk = lc($kk);

			$vv =~ s/^\s+//;
			$vv =~ s/\s+$//;

			$alltabs->{$tname}->{with}->{$kk} = $vv;

		}
		
		die "no relid for table $tname"
			unless (exists($alltabs->{$tname}->{with}->{relid}));

		# build the common comment tag 
		# (of the form /* relation id: nnn - tablename date */ ) 
		# This comment is affixed to generated code (and is used as a
		# marker to find these entries when we need to cleanout
		# duplicates)

		$alltabs->{$tname}->{relid_comment_tag} = 
			"/* relation id: " .
			$alltabs->{$tname}->{with}->{relid} .
			" - $tname " .
			yyyy_mm_dd() . " */\n";

		if (exists($alltabs->{$tname}->{with}->{shared}))
		{
			$alltabs->{$tname}->{with}->{shared} =
				($alltabs->{$tname}->{with}->{shared} =~ m/t|true|y|yes|1/i);
		}
		else
		{
			$alltabs->{$tname}->{with}->{shared} = 0;
		}

		if (exists($alltabs->{$tname}->{with}->{bootstrap}))
		{
			$alltabs->{$tname}->{with}->{bootstrap} =
				($alltabs->{$tname}->{with}->{bootstrap} =~ m/t|true|y|yes|1/i);
		}
		else
		{
			$alltabs->{$tname}->{with}->{bootstrap} = 0;
		}

		if (exists($alltabs->{$tname}->{with}->{oid}))
		{
			$alltabs->{$tname}->{with}->{oid} =
				($alltabs->{$tname}->{with}->{oid} =~ m/t|true|y|yes|1/i);
		}
		else
		{
			$alltabs->{$tname}->{with}->{oid} = 1;
		}

		# build a camel-case string for the tablename
		unless (exists($alltabs->{$tname}->{with}->{camelcase}))
		{
			my $cc1 = $tname;
			my $isgp = 0;

			if ($cc1 =~ m/^(gp)\_/i)
			{
				# treat camel-case name special for Gp tables...
				$cc1 =~ s/^...//;
				$isgp = 1;
			}
			if ($cc1 =~ m/^(pg)\_/i)
			{
				# remove prefix
				$cc1 =~ s/^...//;
			}
			$cc1 = ucfirst($cc1);
			$cc1 = "Gp" . $cc1 if ($isgp);
			$alltabs->{$tname}->{with}->{camelcase} = $cc1;
		}

		if (exists($alltabs->{$tname}->{with}->{reltype_oid}))
		{
			my $relid = $alltabs->{$tname}->{with}->{relid};

			die "table $tname has a dynamic reltype oid -- cannot redefine to static!!"
				if (exists($dynamic_reltype_h{$relid}));
		}
		else
		{
			my $relid = $alltabs->{$tname}->{with}->{relid};

			die "table $tname must have a static reltype oid"
				unless (exists($dynamic_reltype_h{$relid}));
		}

	} # end if with


#	print "bigstr: ", $bigstr, "\n";

	parsecols($alltabs, $tname, $bigstr);

#	print "bigstr: ", $bigstr, "\n";

	# CONTENT checks: 
	# NOTE: no code generation - only for validation
	if (exists($alltabs->{$tname}->{with}->{content}))
	{
		die "unknown content type for $tname: $alltabs->{$tname}->{with}->{content}"
			unless ($alltabs->{$tname}->{with}->{content} =~
					m/MASTER\_ONLY|PERSISTENT|SEGMENT\_LOCAL/);
	}

	# TOAST table checks

	if (exists($alltabs->{$tname}->{with}->{toast_oid}))
	{
		die "toast table for $tname must have index (toast_index)"
			unless (exists($alltabs->{$tname}->{with}->{toast_index}));
	}
	if (exists($alltabs->{$tname}->{with}->{toast_index}))
	{
		die "toast index for $tname must have table (toast_oid)"
			unless (exists($alltabs->{$tname}->{with}->{toast_oid}));
	}
	if (exists($alltabs->{$tname}->{with}->{toast_oid}))
	{
		die "toast table for $tname must have static reltype (toast_reltype)"
			unless (exists($alltabs->{$tname}->{with}->{toast_reltype}) ||
					exists($toast_reltype_exception_h{$tname}));
	}
	if (exists($alltabs->{$tname}->{with}->{toast_reltype}))
	{
		die "toast table for $tname has dynamic reltype -- cannot redefine to static!!"
			if (exists($toast_reltype_exception_h{$tname}));

		die "toast table static reltype for $tname must have table (toast_oid)"
			unless (exists($alltabs->{$tname}->{with}->{toast_oid}));
	}

	if (exists($alltabs->{$tname}->{cols}) &&
		defined($alltabs->{$tname}->{cols}) &&
		# ignore pre-existing exception tables
		(!exists($toast_tab_exception_h{$tname})))
	{
		my $badmsg = "";

		for my $col1 (@{$alltabs->{$tname}->{cols}})
		{
			my $cname = $col1->{colname};
			my $ctype = $col1->{sqltype};

			# aclitem arrays are ok
			next if ($ctype =~ m/aclitem/);

			if (($ctype =~ m/\[.*\]/) ||
				($ctype =~ /text/))
			{
				last # table has a toast table - ok!
					if (exists($alltabs->{$tname}->{with}->{toast_oid}));

				$badmsg .= "table $tname needs toast table (toast_oid) for column $cname of type $ctype\n";
			}
		} # end for
		die $badmsg if (length($badmsg));

	}

	return $tname;

} # end parsetab


sub parsetabdef
{
	my ($alltabs, $bigstr, $filnam) = @_;
	my @keys;

	# fix for case of semicolons in comments
	{
		my @ll;
		my @lins = split(/\n/, $bigstr);

		for my $lin (@lins)
		{
			# replace semicolon in comment with "dummy semi"
			if ($lin =~ m/\-\-.*\;/)
			{
				$lin =~ s/\;/TIDY_DUMMY_SEMI/g;
			}
			push @ll, $lin;
		}
		$bigstr = join("\n", @ll);
	}

	my @statements = split(';', $bigstr);

#	print Data::Dumper->Dump(\@statements), "\n";

	for my $stat (@statements)
	{
		# replace dummy semicolons with the real thing
		$stat =~ s/TIDY\_DUMMY\_SEMI/;/gm;

		if ($stat =~ m/^\s*create\s+table/is)
		{
			push @keys, parsetab($alltabs, $stat, $filnam);
		}
		elsif ($stat =~ m/^\s*create\s+(unique\s+)?index/is)
		{
			parseindex($alltabs, $stat);
		}
#		elsif ($stat =~ m/^\s*alter\s+table\s+\w+add\s+fk/is)
		elsif ($stat =~ m/^\s*alter\s+table/is)
		{
			parsefk($alltabs, $stat, $filnam);
		}
	}

	return @keys;

} # end parsetabdef

# expects val1, len1, val2, len2
# where length values are based on printed offset, not length(val), ie
# embedded tabs are counted.
sub tabalign
{
	my ($tabwidth, $collist) = @_;
	my $tabstr = $glob_tabstr;

	my $maxlen = 0;
	for my $coldef (@{$collist})	
	{
		die "bad coldef: " . Data::Dumper->Dump([$coldef])
			unless (scalar(@{$coldef}) > 3);

		$maxlen = $coldef->[1] if ($coldef->[1] > $maxlen);
	}

	# find the tab position for the second column
	my $col2tab = (POSIX::ceil($maxlen / $tabwidth)) * $tabwidth;

	$col2tab++ if ($col2tab == $maxlen);

#	print $maxlen, " " , $col2tab, "\n";

#	print Data::Dumper->Dump($collist), "\n";

	for my $ii (0..(scalar(@{$collist})-1))
	{
#		print Data::Dumper->Dump($collist->[$ii]), "\n";

		my $val1 = shift @{$collist->[$ii]};
		my $len1 = shift @{$collist->[$ii]};
		my $val2 = shift @{$collist->[$ii]};
		my $len2 = shift @{$collist->[$ii]};

		my $newval = $val1;

		if ($len1 < $col2tab)
		{
			my $mod1 = $len1 % $tabwidth;

#			print "mod: $mod1\n";

			if ($mod1)
			{
				$len1 += ($tabwidth - $mod1);
				if ($glob_faketab)
				{
					$val1 .= " " x ($tabwidth - $mod1);
				}
				else
				{
					$val1 .= $tabstr;
				}
			}
		}

		while ($len1 < $col2tab)
		{
			$len1 += $tabwidth;
			$val1 .= $tabstr;
		}

		unshift @{$collist->[$ii]}, $len1 + $len2 ;
		unshift @{$collist->[$ii]}, $val1 . $val2;		
	} # end for ii

#	print Data::Dumper->Dump($collist), "\n";

	return $collist;
}

sub simpletabalign
{
	my ($col1, $col2) = @_;

	my $colitem = [];

	push @{$colitem}, $col1;
	push @{$colitem}, length($col1);

	push @{$colitem}, $col2;
	push @{$colitem}, length($col2);

	my $collist = tabalign($glob_tabwidth, [$colitem]);

	return $collist->[0]->[0];
}

# take a table-format string (columns separated by "|", rows separated
# by newline) and return an array with a single, formatted string row
sub tabalignstr
{
	my $str = shift;

	my $flist = [];

	my @lines = split(/\n/, $str);

	return $flist
		unless scalar(@lines);

	for my $lin (@lines)
	{
		my @foo = split(/\|/, $lin);

		last
			unless (scalar(@foo));

		my $flitem = [];
		
		for my $itm (@foo)
		{
			$itm = ""
				unless (defined($itm));
			push @{$flitem}, $itm, length($itm);
		}
		push @{$flist}, $flitem;
	}

  L_bigloop:
	while (1)
	{
		for my $coldef (@{$flist})
		{
			last L_bigloop
				unless (scalar(@{$coldef}) > 3);
		}

		$flist = tabalign($glob_tabwidth, $flist);
	}

	my @itmlst;
	for my $itm (@{$flist})
	{
		push @itmlst, shift(@{$itm});
	}

	return \@itmlst;

} # end tabalignstr

# convert a column name to an "Anum_" column id
sub anum_key
{
	my ($tname, $wkey) = @_;

	my $atname = $tname;
	
	# XXX XXX XXX XXX XXX XXX XXX XXX XXX XXX XXX XXX 
	# XXX XXX XXX XXX XXX XXX XXX XXX XXX XXX XXX XXX 
	# XXX XXX: stupid last_operation/shoperation fixup
	if ($tname =~ m/pg\_stat\_last_(sh)?operation/)
	{
		$atname =~ s/eration$//;
		$atname =~ s/stat\_last\_/statlast/;
	}

	if ($tname =~ m/gp_distribution_policy/)
	{
		$atname =~ s/distribution\_//;
	}
	# XXX XXX XXX XXX XXX XXX XXX XXX XXX XXX XXX XXX 
	# XXX XXX XXX XXX XXX XXX XXX XXX XXX XXX XXX XXX 

	return "Anum_" . $atname . "_" . $wkey;
}

sub struct_form_tname
{
	my $tname = shift;

	my $atname = $tname;
	
	# XXX XXX XXX XXX XXX XXX XXX XXX XXX XXX XXX XXX 
	# XXX XXX XXX XXX XXX XXX XXX XXX XXX XXX XXX XXX 
	# XXX XXX: stupid last_operation/shoperation fixup
	if ($tname =~ m/pg\_stat\_last_(sh)?operation/)
	{
		$atname =~ s/eration$//;
		$atname =~ s/stat\_last\_/statlast/;
			}
	# XXX XXX XXX XXX XXX XXX XXX XXX XXX XXX XXX XXX 
	# XXX XXX XXX XXX XXX XXX XXX XXX XXX XXX XXX XXX 

	return "Form_" . $atname;
}

sub formatcolconst
{
	my ($tnamepref, $allcol) = @_;

	my $collist = [];

	{
		my $colitem = [];
		my $cdef;

		$cdef = "#define Natts_" . $tnamepref;

		push @{$colitem}, $cdef;
		push @{$colitem}, length($cdef) ;
		push @{$colitem}, scalar(@{$allcol});
		push @{$colitem}, length(scalar(@{$allcol}));
			
		push @{$collist}, $colitem;
	}

	my $ii = 1;

	for my $coldef (@{$allcol})
	{
		my $colitem = [];
		my $cdef;

		$cdef = "#define " . anum_key($tnamepref, $coldef->{colname});

		push @{$colitem}, $cdef;
		push @{$colitem}, length($cdef) ;

		push @{$colitem}, $ii;
		push @{$colitem}, length($ii);

		$ii++;
			
		push @{$collist}, $colitem;
	}

	$collist = tabalign($glob_tabwidth, $collist);

	my $bigstr = "";

	for my $coldef (@{$collist})
	{
		my $st1 = shift @{$coldef};

		$bigstr .= $st1 . "\n";
	}

	return $bigstr;

} # end formatcolconst

sub formatcols
{
	my $allcol = shift;

	my $collist = [];

	my $bigstr = "";

	# formatting trick: col1 is prefixed by a tab, so increase its
	# length by tabwidth.  col2 is the column name followed by a
	# semicolon.  And col3 is the optional comment.  Build an array of
	# items of [val1, len1, val2, len2, val3, len3]
	#
	# tabalign will shift off the first 4 entries and append the values
	# with embedded tabs to align them, nicely, then unshift the
	# append values and combined length back into original array,
	# resulting in:
	#  ["val1<tabs>val2", len1+<tab widths>+len2, val3, len3]
	#
	# So now we just feed this array back into tabalign to get:
	#  ["val1<tabs>val2<tabs>val3", len1+<tab widths>+len2+<tab widths>+len3]

	for my $coldef (@{$allcol})
	{
		my $colitem = [];
		my $coltype = $coldef->{ctype};
		my $colname = $coldef->{colname};

		# fix arrays of types -- the array suffix moves from the
		# typename to the column name
		if ($coltype =~ m/\[1\]/)
		{
			$coltype =~ s/\[1\]//;
			$colname .= "[1]";
		}

		push @{$colitem}, $glob_tabstr . $coltype;
		push @{$colitem}, length($coltype) + $glob_tabwidth;

		push @{$colitem}, $colname . ";";
		push @{$colitem}, length($colname) + 1;

		if (exists($coldef->{postcomment}))
		{
			my $pc = $coldef->{postcomment};

			$pc =~ s/^\-\-/\/\*/;
			$pc .= " \*\/";

			push @{$colitem}, $pc;
			push @{$colitem}, length($pc);
		}
		else
		{
			push @{$colitem}, "";
			push @{$colitem}, 0;
		}

		push @{$collist}, $colitem;
	}

	$collist = tabalign($glob_tabwidth, $collist);
	$collist = tabalign($glob_tabwidth, $collist);

	for my $ii (0..(scalar(@{$collist})-1))
	{
		my $coldef1 = $allcol->[$ii];
		my $coldef2 = $collist->[$ii];
		my $st1 = shift @{$coldef2};

		if (1 && (exists($coldef1->{precomment})))
		{
			my $pc = $coldef1->{precomment};
			my $bComment = 0;

			$bComment = ($pc =~ m/^\s*\-\-/m);

			if ($bComment)
			{
				# make everything into a single-line comment
				$pc =~ s/^\s*\-\-(.*)/$glob_tabstr\/\*$1 \*\//gm;

				# find adjacent single-line comments by looking for
				# "*/\n/*" and merge them together
				$pc =~ s/\*\/\s*(\n\s*)\/\*/$1 \*/gm;

				$pc .= "\n";
			}

			# remove the blank lines
			$pc =~ s/\*\*TK\_BLANK\_LINE\*\*//gm;

			$bigstr .= $pc;
		}

		$bigstr .= $st1 . "\n";
	}

	return $bigstr;

} # end formatcols


sub formatTZcomment
{
	my $colname = shift;

	my $bigstr = <<"EOF_bigstr";
/*
 * The CATALOG definition has to refer to the type of MYTYPENAME as
 * "timestamptz" (lower case) so that bootstrap mode recognizes it.  But
 * the C header files define this type as TimestampTz.	Since the field is
 * potentially-null and therefore cannot be accessed directly from C code,
 * there is no particular need for the C struct definition to show the
 * field type as TimestampTz --- instead we just make it Datum.
 */

#define timestamptz Datum

EOF_bigstr

	$bigstr =~ s/MYTYPENAME/$colname/;
	return $bigstr;
}

sub formatTMcomment
{
	my $colname = shift;

	my $bigstr = <<"EOF_bigstr";
/*
 * The CATALOG definition has to refer to the type of MYTYPENAME as
 * "time" (lower case) so that bootstrap mode recognizes it.  But
 * the C header files define this type as TimeADT.	So we use a sleazy trick.
 *
 */

#define time TimeADT

EOF_bigstr

	$bigstr =~ s/MYTYPENAME/$colname/;
	return $bigstr;
}

# print YEARMONTHDAY as 8 digit string
sub yyyy_mm_dd
{
	my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) =
		localtime();

	# adjust year and month to normal
	$year += 1900;
	$mon++; 
	
	# format with lead zero if necessary
	$mon  = sprintf("%02d", $mon);
	$mday = sprintf("%02d", $mday);

	return $year.$mon.$mday;

}

sub fix_issharedrelation_function
{
	my $alltabs = shift;
	my $bigstr = "";
	my $toaststr = "";
	my %relidh;

	# nicer to sort by relid (vs tablename)
	while (my ($jj, $ww) = each(%{$alltabs}))
	{
		$relidh{$ww->{with}->{relid}} = $jj;
	}

#	for my $kk (sort (keys (%{$alltabs})))
	for my $jj (sort {$a <=> $b} (keys (%relidh)))
	{
		my $kk = $relidh{$jj};
		my $vv = $alltabs->{$kk};
		my $relid = $vv->{with}->{relid};
		my $rct   = $vv->{relid_comment_tag}; # standard comment

		next
			unless ($vv->{with}->{shared});

#		print "iss: ", Data::Dumper->Dump([$vv]), "\n";

		$bigstr .= "\n\n";
		$bigstr .= $rct;
		$bigstr .= "relationId == " . $vv->{CamelCaseRelationId} . " || ";
		$bigstr .= "\n";

		# build entries for Toast table and toast index
		if (exists($vv->{CamelCaseToastTab}))
		{
			$toaststr .= "\n\n";
			$toaststr .= $rct;
			$toaststr .= "relationId == " . $vv->{CamelCaseToastTab} . " || ";
			$toaststr .= "\n";
			$toaststr .= "relationId == " . $vv->{CamelCaseToastInd} . " || ";
			$toaststr .= "\n";
		}

	}

	my $indstr = "";

	for my $jj (sort {$a <=> $b} (keys (%relidh)))
	{
		my $kk = $relidh{$jj};
		my $vv = $alltabs->{$kk};
		my $relid = $vv->{with}->{relid};
		my $rct   = $vv->{relid_comment_tag}; # standard comment

		next
			unless ($vv->{with}->{shared});

		next unless (exists($vv->{indexes}));

		$indstr .= "\n\n";

		for my $ind (@{$vv->{indexes}})
		{
			$indstr .= $rct;
			$indstr .= "relationId == " . $ind->{CamelCaseIndexId} . " || " ;
			$indstr .= "\n";
		}
		$indstr .= "\n\n";
	}

	return [$bigstr, $indstr, $toaststr];
} # end fix_issharedrelation_function

sub bootparse_str
{
	my ($ccrelid, $ucreltypid) = @_;

	my $bigstr = <<"EOF_bigstr";
SEVENTABScase CAMELCASERELATIONID:
EIGHTTABStypid = UPPERCASERELTYPID;
EIGHTTABSbreak;
EOF_bigstr

	my $seventabs = $glob_tabstr x 7;
	my $eighttabs = $glob_tabstr x 8;

	$bigstr =~ s/SEVENTABS/$seventabs/gm;
	$bigstr =~ s/EIGHTTABS/$eighttabs/gm;

	$bigstr =~ s/CAMELCASERELATIONID/$ccrelid/gm;
	$bigstr =~ s/UPPERCASERELTYPID/$ucreltypid/gm;

	return $bigstr;
}

sub fix_bootparse
{
	my $alltabs = shift;
	my $bigstr = "";
	my %relidh;

	# nicer to sort by relid (vs tablename)
	while (my ($jj, $ww) = each(%{$alltabs}))
	{
		$relidh{$ww->{with}->{relid}} = $jj;
	}

#	for my $kk (sort (keys (%{$alltabs})))
	for my $jj (sort {$a <=> $b} (keys (%relidh)))
	{
		my $kk = $relidh{$jj};
		my $vv = $alltabs->{$kk};
		my $relid = $vv->{with}->{relid};
		my $rct   = $vv->{relid_comment_tag}; # standard comment

		next
			unless (exists($vv->{with}->{reltype_oid}));

		$bigstr .= "\n\n";
		$bigstr .= $rct;
		$bigstr .=  bootparse_str($vv->{CamelCaseRelationId}, 
								  $vv->{UppercaseReltypeOid});
		$bigstr .= "\n\n";
	}

	return $bigstr;

} # end fix_bootparse

sub toastbootstrap_str
{
	my ($cctoastid, $ucreltypid) = @_;

	my $bigstr = <<"EOF_bigstr";
TWOTABScase CAMELCASETOASTID:
THREETABStypid = UPPERCASERELTYPID;
THREETABSbreak;
EOF_bigstr

	my $twotabs   = $glob_tabstr x 2;
	my $threetabs = $glob_tabstr x 3;

	$bigstr =~ s/TWOTABS/$twotabs/gm;
	$bigstr =~ s/THREETABS/$threetabs/gm;

	$bigstr =~ s/CAMELCASETOASTID/$cctoastid/gm;
	$bigstr =~ s/UPPERCASERELTYPID/$ucreltypid/gm;

	return $bigstr;
}

sub fix_toastbootstrap
{
	my $alltabs = shift;
	my $bigstr = "";
	my %relidh;

	# nicer to sort by relid (vs tablename)
	while (my ($jj, $ww) = each(%{$alltabs}))
	{
		$relidh{$ww->{with}->{relid}} = $jj;
	}

#	for my $kk (sort (keys (%{$alltabs})))
	for my $jj (sort {$a <=> $b} (keys (%relidh)))
	{
		my $kk = $relidh{$jj};
		my $vv = $alltabs->{$kk};
		my $rct   = $vv->{relid_comment_tag}; # standard comment

		next
			unless (exists($vv->{with}->{toast_reltype}));

		$bigstr .= "\n\n";
		$bigstr .= $rct;
		$bigstr .=  toastbootstrap_str($vv->{CamelCaseToastTab}, 
									   $vv->{UppercaseToastReltypeOid});
		$bigstr .= "\n\n";
	}

	return $bigstr;

} # end fix_toastbootstrap

sub get_all_filenames
{
	my ($alltabs, $h_allfiles) = @_;
	
	# store the basename of the file in a hash
	for my $kk (sort (keys (%{$alltabs})))
	{
		my ($vol, $dirs, $basename) = 
			File::Spec->splitpath($alltabs->{$kk}->{filename});
		
		# skip exceptions
		next if (exists($allfiles_exception_h{$basename}));

		$h_allfiles->{$basename} = $kk;
	}

} # end get_all_filenames


sub formattypedata
{
	my ($oid, $tname, $reltype_oid) = @_;

	my $bigstr = <<'EOF_bigstr';
DATA(insert OID = RELTYPE_OID ( TABLENAME	    PGNSP PGUID -1 f c t \054 THEE_OID 0 record_in record_out record_recv record_send - d x f 0 -1 0 _null_ _null_ ));	
EOF_bigstr

$bigstr =~ s/TABLENAME/$tname/gm;
$bigstr =~ s/RELTYPE\_OID/$reltype_oid/gm;
$bigstr =~ s/THEE\_OID/$oid/gm;

return $bigstr;

}
sub formattoasttypedata
{
	my ($oid, $tname, $reltype_oid, $toast_oid, $toast_reltype) = @_;

	my $bigstr = <<'EOF_bigstr';
DATA(insert OID = TOAST_RELTYPE (pg_toast_THEE_OID TOASTNSP PGUID -1 f c t \054 TOAST_OID 0 record_in record_out record_recv record_send - d x f 0 -1 0 _null_ _null_));
EOF_bigstr

$bigstr =~ s/TABLENAME/$tname/gm;
$bigstr =~ s/RELTYPE\_OID/$reltype_oid/gm;
$bigstr =~ s/THEE\_OID/$oid/gm;
$bigstr =~ s/TOAST\_OID/$toast_oid/gm;
$bigstr =~ s/TOAST\_RELTYPE/$toast_reltype/gm;

return $bigstr;

}

sub formattypes
{
	my $alltabs = shift;
	my $bigstr = "";
	my %relidh;

	# nicer to sort by relid (vs tablename)
	while (my ($jj, $ww) = each(%{$alltabs}))
	{
		$relidh{$ww->{with}->{relid}} = $jj;
	}

#	for my $kk (sort (keys (%{$alltabs})))
	for my $jj (sort {$a <=> $b} (keys (%relidh)))
	{
		my $kk = $relidh{$jj};
		my $vv = $alltabs->{$kk};
		my $relid = $vv->{with}->{relid};
		my $rct   = $vv->{relid_comment_tag}; # standard comment

		next
			unless (exists($vv->{with}->{reltype_oid}));

		my $reltype_oid = $vv->{with}->{reltype_oid};

		my $uc_tname = uc($kk);

		$vv->{UppercaseReltypeOid} = $uc_tname . "_RELTYPE_OID";

		$bigstr .= $rct;
		$bigstr .= formattypedata($relid, $kk, $reltype_oid);
		$bigstr .= 
			"#define " . $uc_tname . "_RELTYPE_OID " . $reltype_oid . "\n\n\n";

		# do TOAST
		next
			unless (exists($vv->{with}->{toast_reltype}) &&
					exists($vv->{with}->{toast_oid}));

		$vv->{UppercaseToastReltypeOid} = $uc_tname . "_TOAST_RELTYPE_OID";
		$bigstr .= $rct;
		$bigstr .= formattoasttypedata($relid, $kk, $reltype_oid, 
									   $vv->{with}->{toast_oid},
									   $vv->{with}->{toast_reltype}
			);
		$bigstr .= 
			"#define " . $vv->{UppercaseToastReltypeOid} . " " .
			$vv->{with}->{toast_reltype} . "\n\n\n";
	}

	return $bigstr;
}

sub formatindexes
{
	my $alltabs = shift;
	my $bigstr = "";
	my %relidh;

	# nicer to sort by relid (vs tablename)
	while (my ($jj, $ww) = each(%{$alltabs}))
	{
		$relidh{$ww->{with}->{relid}} = $jj;
	}

#	for my $kk (sort (keys (%{$alltabs})))
	for my $jj (sort {$a <=> $b} (keys (%relidh)))
	{
		my $kk = $relidh{$jj};
		my $vv = $alltabs->{$kk};
		my $relid = $vv->{with}->{relid};
		my $rct   = $vv->{relid_comment_tag}; # standard comment

		next unless (exists($vv->{indexes}));

		for my $ind (@{$vv->{indexes}})
		{
			my $iname;
			my $icolist;
			my $camelname;

			$iname = $kk;
			$icolist = "";
			$camelname = $vv->{with}->{camelcase};

			for my $cols (@{$ind->{cols}})
			{
				$iname .= "_" . $cols->[0];
				$camelname .= ucfirst($cols->[0]);

				if (length($icolist))
				{
					$icolist .= ", ";
				}
				$icolist .= $cols->[0] . " " . $cols->[1];
			}
			$iname .= "_index";

			# if indexname was supplied, use it instead
			if (exists($ind->{indexname}))
			{
				$iname = $ind->{indexname};
			}

			if ($ind->{unique})
			{
				$bigstr .= $rct;
				$bigstr .= "DECLARE_UNIQUE_INDEX(";
			}
			else
			{
				$bigstr .= $rct;
				$bigstr .= "DECLARE_INDEX(";
			}
			my $indexid = $ind->{indexid};

			$bigstr .=
				$iname . ", " . "$indexid, on " . $kk . 
				" using btree(" . $icolist . "));\n";

			# use user-supplied CamelCase name if it exists
			if (exists($ind->{with}->{camelcase}))
			{
				$camelname = $ind->{with}->{camelcase};
			}

			$ind->{CamelCaseIndexId} = $camelname . "IndexId";

			my $ccdef = "#define " . $camelname . "IndexId";

			$bigstr .= simpletabalign($ccdef, $indexid) . "\n\n\n";

		}
	}

	return $bigstr;
} # end formatindexes

sub format_syscache_cacheinfo
{
	my $cinfo = shift;

	my $vv = $cinfo;

# example:
#
#	{AggregateRelationId,		/* AGGFNOID */
#		AggregateAggfnoidIndexId,
#		1,
#		{
#			Anum_pg_aggregate_aggfnoid,
#			0,
#			0,
#			0
#		},
#		32
#	},


	my $bigstr = <<"EOF_bigstr";
FIRSTLINE
TWOTABSINDID,
TWOTABSCOLCNT,
TWOTABS{
ALLCOLS
TWOTABS},
TWOTABSNBUCKETS
EOF_bigstr

	my $twotabs = $glob_tabstr x 2;

	$bigstr =~ s/TWOTABS/$twotabs/gm;

	# slight goofiness with end curly brace to fix emacs formatting...
	$bigstr .= "\t" . '}';

	$bigstr =~ s/FIRSTLINE/$vv->{firstline}/;
	$bigstr =~ s/INDID/$vv->{indid}/;
	$bigstr =~ s/NBUCKETS/$vv->{nbuckets}/;

	my $colcnt = scalar(@{$vv->{cols}});
	$bigstr =~ s/COLCNT/$colcnt/;

	# build array of length 4, with zeroes in "null" positions
	my $cols = [];
	for my $ii (0..3)
	{
		if ($ii >= $colcnt)
		{
			push @{$cols}, '0';
			next;
		}

		my $c1 = $vv->{cols}->[$ii];

		if (($c1 eq 'oid')
			&& $vv->{with_oid})
		{
			push @{$cols}, 'ObjectIdAttributeNumber';
		}
		else
		{
			push @{$cols}, anum_key($vv->{relname}, $c1);
		}
	}

	my $allcols = $glob_tabstr x 3;
	$allcols .= join(",\n\t\t\t", @{$cols});

	$bigstr =~ s/ALLCOLS/$allcols/;

	return $bigstr;

} # end format_syscache_cacheinfo

sub fixup_syscache_cfile
{
	my ($alltabs, $cacheh, $fullcname, $tmpcname, 
		$gen_hdr_str) = @_;

	my $bigstr = "";

	$bigstr = $gen_hdr_str;

	my $whole_file;
	
	{
        # $$$ $$$ undefine input record separator (\n)
        # and slurp entire file into variable
		
        local $/;
        undef $/;
		
		my $fh;

		open $fh, "< $fullcname" 
			or die "cannot open $fullcname: $!";

		$whole_file = <$fh>;
		
		close $fh;
	}

	# build array of strings of "first lines", eg
	# "{AggregateRelationId,  /* AGGFNOID */"
	# using tabalignstr
	my $tab1 = "";
	for my $kk (sort(keys(%{$cacheh})))
	{
		my $vv = $cacheh->{$kk};

		$tab1 .= "|{" . $vv->{relid} . ",|/* " . $kk . " */\n";
	}

	my $firstlin = tabalignstr($tab1);

	# format every cacheinfo entry in alphabetical order
	my @out1;
	for my $kk (sort(keys(%{$cacheh})))
	{
		my $vv = $cacheh->{$kk};

		$vv->{firstline} = shift @{$firstlin};

		push @out1, format_syscache_cacheinfo($vv);
	}

	# join the formatted entries
	$bigstr .= "\n" .
		"static const struct cachedesc cacheinfo[] = {\n" .
		join(",\n", @out1) . "\n};\n";

	my $prefx = quotemeta('TIDYCAT_BEGIN_CODEGEN');
	my $suffx = quotemeta('TIDYCAT_END_CODEGEN');

	my @zzz = ($whole_file =~ 
			   m/^\s*\/\*\s*$prefx\s*\s*$(.*)^\s*\/\*\s*$suffx\s*\*\/\s*$/ms);

	die "bad target: $whole_file"
		unless (scalar(@zzz));

	my $rex = $zzz[0];

	# replace carriage returns first, then quotemeta, then fix CR again...
	$rex =~ s/\n/SLASHNNN/gm;
	$rex = quotemeta($rex);
	$rex =~ s/SLASHNNN/\\n/gm;
	
	# substitute the new generated definitions for the prior
	# generated definitions in the target file
	$whole_file =~ s/$rex/$bigstr/ms;

	my $outi;
	
	open $outi, "> $tmpcname"
		or die "cannot open $tmpcname for write: $!";
	
	# rewrite the target file
	print $outi $whole_file;
				
	close $outi;

} # end fixup_syscache_cfile

sub fixup_syscache
{
	my ($alltabs, $fullhname, $tmphname, $fullcname, $tmpcname,
		$verzion, $nnow) = @_;

	my $bigstr = "";
	my %cacheh;

	my $gen_hdr_str = "\n * WARNING: DO NOT MODIFY THE FOLLOWING SECTION: \n" .
	" *  Generated by " . $verzion . "\n" . 
	" *  on " . $nnow . "\n*/\n";

	$bigstr = $gen_hdr_str;

	my $whole_file;
	
	{
        # $$$ $$$ undefine input record separator (\n)
        # and slurp entire file into variable
		
        local $/;
        undef $/;
		
		my $fh;

		open $fh, "< $fullhname" 
			or die "cannot open $fullhname: $!";

		$whole_file = <$fh>;
		
		close $fh;
	}

	for my $kk (keys (%{$alltabs}))
	{
		my $vv = $alltabs->{$kk};
		my $relid = $vv->{with}->{relid};
		my $rct   = $vv->{relid_comment_tag}; # standard comment

#		print $kk, "\n";

		next unless (exists($vv->{indexes}));

		for my $ind (@{$vv->{indexes}})
		{
			my $iname;
			my $icolist;
			my $camelname;

			next unless (exists($ind->{with}->{syscacheid}));

			die "$kk: invalid syscacheid for non-unique index"
				unless ($ind->{unique});

			my $cols = [];

			# get colnames
			for my $c1 (@{$ind->{cols}})
			{
				push @{$cols}, $c1->[0];
			}

			my $with_oid = (
				(exists($vv->{with}->{oid})) &&
				($vv->{with}->{oid}));

			$cacheh{$ind->{with}->{syscacheid}} = {
				relname => $kk,
				relid => $vv->{CamelCaseRelationId},
				indid => $ind->{CamelCaseIndexId},
				nbuckets => $ind->{with}->{syscache_nbuckets},
				with_oid => $with_oid,
				cols => $cols
			};
		}
	}

	# XXX XXX: not enough syscacheid ?
	return
		if (scalar(keys(%cacheh)) <= 2);

	{
		my @sid = sort(keys(%cacheh));

		# label the first and last elements with their integer value
		my $num = scalar(@sid) - 1;

		$sid[0] .= " = 0";
		$sid[-1] .= " = $num";

		$bigstr .= "enum SysCacheIdentifier\n{\n\t" .
			join(",\n\t", @sid) .
			"\n};\n";
	}

	my $prefx = quotemeta('TIDYCAT_BEGIN_CODEGEN');
	my $suffx = quotemeta('TIDYCAT_END_CODEGEN');

	my @zzz = ($whole_file =~ 
			   m/^\s*\/\*\s*$prefx\s*\s*$(.*)^\s*\/\*\s*$suffx\s*\*\/\s*$/ms);

	die "bad target: $whole_file"
		unless (scalar(@zzz));

	my $rex = $zzz[0];

	# replace carriage returns first, then quotemeta, then fix CR again...
	$rex =~ s/\n/SLASHNNN/gm;
	$rex = quotemeta($rex);
	$rex =~ s/SLASHNNN/\\n/gm;
	
	# substitute the new generated definitions for the prior
	# generated definitions in the target file
	$whole_file =~ s/$rex/$bigstr/ms;

	my $outi;
	
	open $outi, "> $tmphname"
		or die "cannot open $tmphname for write: $!";
	
	# rewrite the target file
	print $outi $whole_file;
	
	# now do syscache.c
	fixup_syscache_cfile($alltabs, \%cacheh, $fullcname, $tmpcname, 
						 $gen_hdr_str);

} # end fixup_syscache

sub formattoastheaders
{
	my $alltabs = shift;
	my $bigstr = "";
	my %relidh;

	# nicer to sort by relid (vs tablename)
	while (my ($jj, $ww) = each(%{$alltabs}))
	{
		$relidh{$ww->{with}->{relid}} = $jj;
	}

#	for my $kk (sort (keys (%{$alltabs})))
	for my $jj (sort {$a <=> $b} (keys (%relidh)))
	{
		my $kk = $relidh{$jj};
		my $vv = $alltabs->{$kk};
		my $relid = $vv->{with}->{relid};
		my $rct   = $vv->{relid_comment_tag}; # standard comment

		next unless (exists($vv->{with}->{toast_oid}));

		{
			my $camelname;

			$camelname = $vv->{with}->{camelcase};
			
			# maintain PG/GP prefix for the camelname for TOAST.
			# weird, but true...
			if ($kk =~ m/pg/i)
			{
				$camelname = "Pg" . $camelname
					unless ($camelname =~ m/pg/i);
			}
			elsif ($kk =~ m/gp/i)
			{
				$camelname = "Gp" . $camelname
					unless ($camelname =~ m/gp/i);
			}


			$bigstr .= $rct;
			$bigstr .= "DECLARE_TOAST(";
			$bigstr .=
				$kk . ", " . $vv->{with}->{toast_oid} . ", " .
				$vv->{with}->{toast_index} . ");\n";

			my $toasttabname = $camelname . "ToastTable";

			my $ccdef = "#define " . $toasttabname;

			$bigstr .= simpletabalign($ccdef, 
									  $vv->{with}->{toast_oid}) . "\n";


			my $toastindname = $camelname . "ToastIndex";

			$ccdef = "#define " . $toastindname;

			$bigstr .= simpletabalign($ccdef, 
									  $vv->{with}->{toast_index}) . "\n\n\n";

			$vv->{CamelCaseToastTab} = $toasttabname;
			$vv->{CamelCaseToastInd} = $toastindname;
		}
	}

	return $bigstr;
} # end formattoastheaders

sub formattab
{
	my ($alltabs, $keys) = @_;
	my $bigstr = "";

	for my $kk (@{$keys})
	{
		my $vv = $alltabs->{$kk};
		my $relid = $vv->{with}->{relid};

		$bigstr .= "\n/*\n TidyCat Comments for $kk:\n";
		$bigstr .= "  Table is shared, so catalog.c:IsSharedRelation is " .
			"updated.\n" if $vv->{with}->{shared};
		if ($vv->{with}->{bootstrap})
		{
			$bigstr .= "  Table is a **bootstrap** table.\n";
		}
		if ($vv->{with}->{oid})
		{
			$bigstr .= "  Table has an Oid column.\n";
		}
		else
		{
			$bigstr .= "  Table does not have an Oid column.\n";
		}
		if (exists($vv->{with}->{reltype_oid}))
		{
			$bigstr .= "  Table has static type (see pg_types.h).\n"; 
		}
		else
		{
			$bigstr .= "  Table does not have static type " .
				"(only legal for pre-3.3 tables). \n"; 
		}

		# TOAST comments
		if (exists($vv->{with}->{toast_oid}))
		{
			$bigstr .= "  Table has TOASTable columns,";
			if (exists($toast_reltype_exception_h{$kk}))
			{
				$bigstr .= " but TOAST table does not have static type.\n"; 
			}
			else
			{
				$bigstr .= " and TOAST table has static type.\n"; 
			}
		}
		else
		{
			if (exists($toast_tab_exception_h{$kk}))
			{
				$bigstr .= "  Table has TOASTable columns, but NO TOAST table.\n"; 
			}
		}

		if (exists($vv->{with}->{content}))
		{
			if ($vv->{with}->{content} =~ m/MASTER/)
			{
				$bigstr .= "  Table contents are only maintained on MASTER.\n";
			}
			elsif ($vv->{with}->{content} =~ m/PERSISTENT/)
			{
				$bigstr .= "  Table contents maintain PERSISTENT objects in special way.\n";
			}
			elsif ($vv->{with}->{content} =~ m/SEGMENT/)
			{
				$bigstr .= "  Table contents are local to each SEGMENT.\n";
			}
		}

		$bigstr .= "  Table has weird hack for timestamp column.\n " 
			if (exists($vv->{tzhack}));
		$bigstr .= "  Table has weird hack for time column.\n " 
			if (exists($vv->{tmhack}));

		$bigstr .= "\n*/\n\n";

		$bigstr .= formatTZcomment($vv->{tzhack})
			if (exists($vv->{tzhack}));
		$bigstr .= formatTMcomment($vv->{tmhack})
			if (exists($vv->{tmhack}));

		$bigstr .= getcomment1($kk);

		$vv->{CamelCaseRelationId} = $vv->{with}->{camelcase} . "RelationId" ;

		my $ccdef = '#define ' . $vv->{CamelCaseRelationId};

		$bigstr .= simpletabalign($ccdef, $relid) . "\n\n";
			
		$bigstr .= 'CATALOG('. $kk . ',' . $relid . ')';
		$bigstr .= " BKI_BOOTSTRAP" if $vv->{with}->{bootstrap};
		$bigstr .= " BKI_SHARED_RELATION" if $vv->{with}->{shared};
		$bigstr .= " BKI_WITHOUT_OIDS" unless $vv->{with}->{oid};
		$bigstr .= "\n{";

		$bigstr .= formatcols($vv->{cols});

		# build the Form/FormData strings
		my $tform     = struct_form_tname($kk);
		my $tformdata = struct_form_tname($kk);
		$tformdata =~ s/Form/FormData/;

		$bigstr .= "} " . $tformdata . ";\n";

		$bigstr .= "\n#undef timestamptz\n"
			if (exists($vv->{tzhack}));
		$bigstr .= "\n#undef time\n"
			if (exists($vv->{tmhack}));

		$bigstr .= "\n\n";

		$bigstr .= getcomment2($kk, $tform);

		$bigstr .= "typedef " . $tformdata . " *" . $tform . ";\n";

		$bigstr .= "\n\n";

		$bigstr .= getcomment3($kk);

		$bigstr .= formatcolconst($kk, $vv->{cols});

	}

	return $bigstr;

} # end formattab

sub formattabs
{
	my $alltabs = shift;
	my $bigstr = "";

	for my $kk (sort (keys (%{$alltabs})))
	{
		$bigstr .= formattab($alltabs, [$kk]);
	}

	return $bigstr;

} # end formattabs


sub parsefile
{
	my ($file_name, $finddefs) = @_;

	# if not looking for tidycat definitions, just find the code gen
	# sections

	my $begin_def_str = "TIDYCAT_BEGIN_CODEGEN";
	my $end_def_str = "TIDYCAT_END_CODEGEN";
	
	if (defined($finddefs))
	{
		$begin_def_str = "TIDYCAT_BEGINDEF";
		$end_def_str = "TIDYCAT_ENDDEF";

		# XXX XXX: if only dumping sql definitions, allow "fake" ones...
		if ($glob_glob->{sqldef} || $glob_glob->{dumpdef} 
			|| $glob_glob->{syscache})
		{
			$begin_def_str .= "|TIDYCAT_BEGINFAKEDEF";
			$end_def_str .= "|TIDYCAT_ENDFAKEDEF";

		}

	}


	my $entrylist = [];

	my $bigstr ;

	open my $file_in, "< $file_name" or die "cannot open $file_name: $!";

	my $phase = "begindef"; # begindef, enddef, begingen, endgen

	my $ii = 0;

	my $entry;

	for my $ini (<$file_in>)
	{
		$ii++;
		if ($phase =~ m/enddef/)
		{
			if ($ini =~ m/$end_def_str/)
			{
				if ($ini =~ m/\*\//)
				{
					# look for trailing "*/" 
					$entry->{enddef} = $ii;
				}
				else
				{
					# probably the next line
					$entry->{enddef} = $ii + 1;
				}

				$entry->{def} = $bigstr;

				if (defined($finddefs))
				{
					$phase = "begingen";
				}
				else
				{
					# save the current entry
					push @{$entrylist}, $entry;
					$phase = "begindef"; 
				}

				next;
			}

			# append to the string!!
			$bigstr .= $ini;

		}
		elsif ($phase =~ m/begindef/)
		{
			next unless ($ini =~ m/$begin_def_str/);

			$entry = {begindef => $ii};
			$bigstr = "";
			$phase = "enddef";

			next;
		}
		elsif ($phase =~ m/begingen/)
		{
			# might have new definition and no codegen
			if ($ini =~ m/$begin_def_str/)
			{
				push @{$entrylist}, $entry;

				$entry = {begindef => $ii};
				$bigstr = "";
				$phase = "enddef";

				next;

			}
			elsif ($ini =~ m/TIDYCAT\_BEGIN\_CODEGEN/)
			{
				$entry->{begin_codegen} = $ii;
				$phase = "endgen";

			}
			else
			{
				# stop looking if no generated code within 3 lines...

				if (($ii - $entry->{enddef}) > 3)
				{
					# save the current entry
					push @{$entrylist}, $entry;
					$phase = "begindef"; 
				}
				next;
			}

		}
		elsif ($phase =~ m/endgen/)
		{
			if ($ini =~ m/TIDYCAT\_END\_CODEGEN/)
			{
				$entry->{end_codegen} = $ii;

				# save the current entry
				push @{$entrylist}, $entry;


				$bigstr = "";
				$phase = "begindef";

				
			}
			next;


		}

	} # end for ini

	die "unterminated tidycat definition in $file_name at " .
		$entry->{begindef} if (defined($entry) && ($phase =~ m/enddef/));

	if (defined($entry) && ($phase =~ m/begingen/))
	{
		push @{$entrylist}, $entry;
	}

	close $file_in;

	return $entrylist;
} # end parsefile

sub bufferfile
{
	my $file_name = shift;

	open my $file_in, "< $file_name" or die "cannot open $file_name: $!";

	my $buflist = [];

	for my $ini (<$file_in>)
	{
		push @{$buflist}, $ini;
	}
	close $file_in;

	return $buflist;
}

# We may already have previously generated code for a relation.  Need
# to filter it out to eliminate duplicate definitions.  Note that this
# only works as long as the relid does not change.
sub clean_duplicate_entries
{
	my ($alltabs, $basename, $gen_code) = @_;

	my $db1 = 1;

	print $basename, ":\n\n"
		if ($db1);

	# NOTE: list the names of special files with generated code
	return $gen_code
		unless ($basename =~ m/^(indexing\.h|toasting\.h|toasting\.c|catalog\.c toast|catalog\.c ind|catalog\.c tab|bootparse\.y|pg\_type\.h)$/);

	my $skiplines = 0;

	# skip N lines of generated code for duplicate entries
	$skiplines = 2 
		if ($basename =~ m/indexing/);
	$skiplines = 3 
		if ($basename =~ m/toasting\.h/);
	$skiplines = 3 
		if ($basename =~ m/toasting\.c/);
	$skiplines = 2 
		if ($basename =~ m/pg\_type/);
	$skiplines = 1
		if ($basename =~ m/catalog.*ind/);
	$skiplines = 1
		if ($basename =~ m/catalog.*tab/);
	$skiplines = 2
		if ($basename =~ m/catalog.*toast/);
	$skiplines = 3
		if ($basename =~ m/bootparse/);

	for my $kk (sort (keys (%{$alltabs})))
	{
		my $vv = $alltabs->{$kk};
		my $relid = $vv->{with}->{relid};
		my $rct   = $vv->{relid_comment_tag}; # standard comment

		$rct = '/* relation id: ' . $relid;

		next # no match for current relation
			unless ($gen_code =~ m/$rct\s+/);

		next # don't cleanup unless have lines to skip...
			unless ($skiplines);

		my @foo = split (/\n/, $gen_code);

		my @baz;

		while (scalar(@foo))
		{
			my $lin = shift @foo;

			# iter by line, but skip the line which matches the
			# "relation id" comment
			if ($lin !~ m/$rct\s+/)
			{
				push @baz, $lin;
			}
			else
			{
				print "skip: $lin\n"
					if ($db1);

				# and the subsequent N generated lines after the comment
				for my $ii (1..$skiplines)
				{
#					print "skip: ",
					shift @foo
						if (scalar(@foo));
#					print "\n";
				} # end for
			}
		} # end while foo
			
		$gen_code = join ("\n", @baz);

	} # end for kk

	return $gen_code;

} # end clean_duplicate_entries

sub fixup_generated_code
{
	my ($alltabs, $basename, $fullname, $tmpname, $new_gen_list, 
		$verzion, $nnow,
		$fixup_ref) = @_;

	my $elist_index = 1;

	# catalog.c toast case is third, and index case is second set of
	# definitions
	$elist_index = 3
		if ($basename =~ m/catalog\.c toast/);
	$elist_index = 2
		if ($basename =~ m/catalog\.c ind/);

	return
		unless (scalar(@{$new_gen_list}));

	my ($elist, $buffil);

	# handle toast, then index
	if (defined($fixup_ref) && (2 <= scalar(@{$fixup_ref})))
	{
		$elist = shift @{$fixup_ref};
		$buffil = shift @{$fixup_ref};
	}
	else
	{
		$elist = parsefile($fullname);
		$buffil = bufferfile($fullname);
	}

	# XXX XXX XXX XXX: dump of regen
#	print $basename,":\n", Data::Dumper->Dump($elist), "\n";

	return
		unless (scalar(@{$elist}) >= $elist_index);

	my $entry = $elist->[$elist_index - 1];

	my $bigstr = $entry->{def};

	$bigstr = clean_duplicate_entries($alltabs, $basename, $bigstr);

#	print "bigstr: ", $bigstr, "XXX\n";

	my $genstr = "Generated by " . $verzion;

	$bigstr =~ s/Generated by.*tidycat\.pl\s+version\s+\d+\.(\d)*/$genstr/m;

	$bigstr =~ 
s/($genstr)\n(\s+on\s+)(\w+\s+\w+\s+\d+\s+\d+\:\d+\:\d+\s+\d+)/$1\n$2$nnow/m;

	$bigstr .= "\n" . join("", @{$new_gen_list});

	# remove double-newlines
	$bigstr =~ s/\n\n/\n/gm;

	if (exists($entry->{begindef}))
	{
		splice (@{$buffil}, 
				($entry->{begindef}),
				($entry->{enddef} - $entry->{begindef})-1,
				$bigstr);
	}

	# write the modified file to the temp directory
	open my $file_out, "> " . $tmpname
		or die "Cannot open $tmpname for write";

	# write the modified file to the tmpdir
	print $file_out join("", @{$buffil});
	
	close $file_out;

	return [$elist, $buffil];

} # end fixup_generated_code

sub fixup_pg_tidy
{
	my ($h_allfiles, $fullname, $tmpname,
		$verzion, $nnow) = @_;

	my $bigstr = <<'EOF_bigstr';
/*-------------------------------------------------------------------------
 *
 * pg_tidycat.h
 *	  
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *   http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing,
 * software distributed under the License is distributed on an
 * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 * KIND, either express or implied.  See the License for the
 * specific language governing permissions and limitations
 * under the License.
 *
GENCOM
 *-------------------------------------------------------------------------
 */


EOF_bigstr

	my $gen_hdr_str = " * WARNING: DO NOT MODIFY THIS FILE: \n" .
	" *  Generated by " . $verzion . "\n" . 
	" *  on " . $nnow ;

	$bigstr =~ s/GENCOM/$gen_hdr_str/gm;

	my $prevhdr = `grep include $fullname`;

	my @lines = split(/\n/, $prevhdr);

	
	for my $lin (@lines)
	{
		next unless ($lin =~ m/^\s*\#\s*include\s*\"catalog\//);
		my @foo = ($lin =~ m/^\s*\#\s*include\s*\"catalog\/(.*)\"/);

		my $fname = $foo[0];
		
		next if (exists($allfiles_exception_h{$fname}));

		$h_allfiles->{$fname} = 1;
	}

	# build sorted list of all names in 
	# include "catalog/pg_..." format

	$bigstr .= '#include "catalog/' .
		join("\"\n#include \"catalog/", 
			 sort(keys(%{$h_allfiles}))) . "\"\n";

	# write the modified file to the temp directory
	open my $file_out, "> " . $tmpname
		or die "Cannot open $tmpname for write";

	# write the modified file to the tmpdir
	print $file_out $bigstr;
	
	close $file_out;
	
} # end fixup_pg_tidy

sub fixup_makefile
{
	my ($h_allfiles, $fullname, $tmpname,
		$verzion, $nnow) = @_;

	# here doc to replace tidycat_bki_src defs in Makefile
	my $bigstr = <<'EOF_bigstr';

# TIDYCAT_BEGIN_CODEGEN 
#GENHDRSTR
TIDYCAT_BKI_SRCS := \
	GENSTUFF

# TIDYCAT_END_CODEGEN 
EOF_bigstr

	my $whole_file;

	{
        # $$$ $$$ undefine input record separator (\n)
        # and slurp entire file into variable

        local $/;
        undef $/;

		my $fh;

		open $fh, "< $fullname" or die "cannot open $fullname: $!";

		$whole_file = <$fh>;
		
		close $fh;
	}

	my $prefx = quotemeta('TIDYCAT_BEGIN_CODEGEN');
	my $suffx = quotemeta('TIDYCAT_END_CODEGEN');

	my @zzz = ($whole_file =~ m/^\s*\#\s*$prefx\s*$(.*)^\s*\#\s*$suffx\s*$/ms);

	return 
		unless (scalar(@zzz));

	my @lines = split(/\n/, $zzz[0]);

	for my $lin (@lines)
	{
		next unless ($lin =~ m/\.h/);

		# capture the filename
		my @foo = ($lin =~ m/^\s*(.*)\s*(?:\\)?\s*$/);

		my $fname = $foo[0];
		# trim leading and trailing spaces
		$fname =~ s/\\$//;
		$fname =~ s/^\s*//;
		$fname =~ s/\s*$//;
		
		next if (exists($allfiles_exception_h{$fname}));

		$h_allfiles->{$fname} = 1;
	}

	# build sorted list of all names prefixed with tab and ending with
	# slash, eg:
	#  pg_foo.h \ ...

	my $genstuff = join(" \\\n\t",
						sort(keys(%{$h_allfiles})));

	$bigstr =~ s/GENSTUFF/$genstuff/;

	my $gen_hdr_str = " WARNING: DO NOT MODIFY THE FOLLOWING SECTION: \n" .
		"# Generated by " . $verzion . "\n" . 
		"# on " . $nnow . "\n\n";

	$bigstr =~ s/GENHDRSTR/$gen_hdr_str/;

	# write the modified file to the temp directory
	open my $file_out, "> " . $tmpname
		or die "Cannot open $tmpname for write";

	# replace with modified expression
	$whole_file =~ s/^\s*\#\s*$prefx\s*$(.*)^\s*\#\s*$suffx\s*$/$bigstr/ms;

	# write the modified file to the tmpdir
	print $file_out $whole_file;
	
	close $file_out;
	
} # end fixup_makefile

if (1)
{
# 5000 table
# 6000 indexes

    my $verzion = "unknown";
	$verzion = $glob_glob->{_sleazy_properties}->{version}
	if (exists($glob_glob->{_sleazy_properties}) &&
		exists($glob_glob->{_sleazy_properties}->{version}));

	$verzion = $0 . " version " . $verzion;
	my $nnow = localtime;
	my $gen_hdr_str = "/* TIDYCAT_BEGIN_CODEGEN \n\n";
	$gen_hdr_str .= "   WARNING: DO NOT MODIFY THE FOLLOWING SECTION: \n" .
		"   Generated by " . $verzion . "\n" . 
		"   on " . $nnow . "\n*/\n\n";

	my ($vol0, $dirs0, $base0) = File::Spec->splitpath($0);

	# make sure have a directory for tidypath location
	$dirs0 = File::Spec->rel2abs(".") unless (length($dirs0));

#	print "dirs0: $dirs0\n";

	my @backup_files; # list of files to backup

	my $tmp_indexing = File::Spec->catfile($glob_tmpdir, "indexing.h");
	my $fil_indexing = File::Spec->catfile($dirs0, "indexing.h");
	my $buf_indexing;
	my @arr_indexing;

	push @backup_files, $fil_indexing;

	my $tmp_toastingh = File::Spec->catfile($glob_tmpdir, "toasting.h");
	my $fil_toastingh = File::Spec->catfile($dirs0, "toasting.h");
	my $buf_toastingh;
	my @arr_toastingh;

	push @backup_files, $fil_toastingh;

	my $tmp_pg_type = File::Spec->catfile($glob_tmpdir, "pg_type.h");
	my $fil_pg_type = File::Spec->catfile($dirs0, "pg_type.h");
	my $buf_pg_type;
	my @arr_pg_type;

	push @backup_files, $fil_pg_type;

	# dirs0 should be src/include/catalog
	my @srcdirs = File::Spec->splitdir($dirs0);
	push @srcdirs, "..";
	push @srcdirs, "..";

	# src/backend/catalog
	my $tmp_catalogc = File::Spec->catfile($glob_tmpdir, "catalog.c");
	my $fil_catalogc = File::Spec->catfile(
		File::Spec->catdir(@srcdirs, "backend", "catalog"), "catalog.c");
	my $buf_catalogc;
	my @arr_catalogc;
	my @arr_catalogc_tab;
	my @arr_catalogc_ind;
	my @arr_catalogc_toast;

	push @backup_files, $fil_catalogc;

	my $tmp_toastingc = File::Spec->catfile($glob_tmpdir, "toasting.c");
	my $fil_toastingc = File::Spec->catfile(
		File::Spec->catdir(@srcdirs, "backend", "catalog"), "toasting.c");
	my $buf_toastingc;
	my @arr_toastingc;

	push @backup_files, $fil_toastingc;

	# src/backend/bootstrap
	my $tmp_bootparse = File::Spec->catfile($glob_tmpdir, "bootparse.y");
	my $fil_bootparse = File::Spec->catfile(
		File::Spec->catdir(@srcdirs, "backend", "bootstrap"), "bootparse.y");
	my $buf_bootparse;
	my @arr_bootparse;

	push @backup_files, $fil_bootparse;

	# src/backend/catalog/Makefile
	my $tmp_catmakem = File::Spec->catfile($glob_tmpdir, "Makefile");
	my $fil_catmakem = File::Spec->catfile(
		File::Spec->catdir(@srcdirs, "backend", "catalog"), "Makefile");
	my $buf_catmakem;
	my @arr_catmakem;

	push @backup_files, $fil_catmakem;

	# src/include/catalog/pg_tidycat.h
	my $tmp_pg_tidy = File::Spec->catfile($glob_tmpdir, "pg_tidycat.h");
	my $fil_pg_tidy = File::Spec->catfile($dirs0, "pg_tidycat.h");
	my $buf_pg_tidy;
	my %h_pg_tidy;

	push @backup_files, $fil_pg_tidy;

	# src/include/utils/syscache.h
	my $tmp_syscacheh = File::Spec->catfile($glob_tmpdir, "syscache.h");
	my $fil_syscacheh = File::Spec->catfile( 
		File::Spec->catdir(@srcdirs, "include", "utils"), "syscache.h");

	push @backup_files, $fil_syscacheh;

	# src/backend/utils/cache/syscache.c
	my $tmp_syscachec = File::Spec->catfile($glob_tmpdir, "syscache.c");
	my $fil_syscachec = File::Spec->catfile( 
		File::Spec->catdir(@srcdirs, "backend", "utils", "cache"), 
		"syscache.c");

	push @backup_files, $fil_syscachec;

	my $backup_dir;

	# backup files that may get changed
	unless ($glob_glob->{sqldef} ||
			($glob_glob->{dumpdef}))
	{
		$backup_dir = 
			File::Spec->catfile($glob_tmpdir, "tidycat_backup." . $$);

		die "Cannot open backup directory $backup_dir : $! "
			unless (mkdir($backup_dir));
		
		my $cpargs = join(" ", @backup_files);

		my $cpstat = `cp $cpargs $backup_dir`;

	}

	my %alltabs;

	my @arr_sql; # array of sql statement output
	my %dumptabs_h; # hash of table definitions

	for my $filnam (@ARGV)
	{
		%alltabs = ();

		my ($vol, $dirs, $basename) = File::Spec->splitpath($filnam);

		my $elist = parsefile($filnam, 1); # file has def and codegen sections

		my $buffil = bufferfile($filnam);

		# do it in reverse order to strip out line numbering correctly
		for my $entry (reverse (@{$elist}))
		{
			my $bigstr = $entry->{def};
			
#	print $bigstr, "\n";

			# tricky bit here: might have multiple CREATE table
			# statements in a tidycat definition entry, and multiple
			# tidycat defs in a single file.  So build the list of all
			# table definitions in alltabs, but extract the set of
			# keys (tablenames) for the current entry in parsetabdef,
			# so we can feed that to formattab and generate the code
			# for just the current entry.

			my @keys = parsetabdef(\%alltabs, $bigstr, $filnam);

#	print Data::Dumper->Dump([%alltabs]);

#			print "fff:", formattabs(\%alltabs);

			my $gentext = [];

			push @{$gentext}, split(/(\n)/, $gen_hdr_str);

			push @{$gentext}, split(/(\n)/, formattab(\%alltabs, \@keys));

			push @{$gentext}, "\n\n/* TIDYCAT_END_CODEGEN */\n";

			if (exists($entry->{begin_codegen}))
			{
				splice (@{$buffil}, 
						($entry->{begin_codegen}-1),
						($entry->{end_codegen} - $entry->{begin_codegen}) + 1,
						@{$gentext});
			}
			else
			{

				splice (@{$buffil}, ($entry->{enddef}),
						0,
						@{$gentext});
			}
		} # end for entry

		push @arr_indexing, formatindexes(\%alltabs);

		push @arr_toastingh, formattoastheaders(\%alltabs);
		
		push @arr_pg_type, formattypes(\%alltabs);
		push @arr_toastingc, fix_toastbootstrap(\%alltabs);

		my $tab_ind = fix_issharedrelation_function(\%alltabs);

#		print Data::Dumper->Dump($tab_ind), "\n";

		push @arr_catalogc, $tab_ind;

		if (defined($tab_ind) && scalar(@{$tab_ind}))
		{
			push @arr_catalogc_tab, $tab_ind->[0]
				if (length($tab_ind->[0]));
			push @arr_catalogc_ind, $tab_ind->[1]
				if ((1 < scalar(@{$tab_ind})) && (length($tab_ind->[1])));
			push @arr_catalogc_toast, $tab_ind->[2]
				if ((2 < scalar(@{$tab_ind})) && (length($tab_ind->[2])));
		}

		push @arr_bootparse, fix_bootparse(\%alltabs);
		
		# store the filenames for pg_tidycat and Makefile fixup
		get_all_filenames(\%alltabs, \%h_pg_tidy);

		if ($glob_glob->{sqldef})
		{
			while (my ($jj, $ww) = each(%alltabs))
			{
				if (exists($ww->{tabdef_text})
					&& length($ww->{tabdef_text})
					&& ($ww->{tabdef_text} =~ m/CREATE\s+TABLE/i))
				{
					push @arr_sql, $ww->{tabdef_text};
				}
			}

			next; # don't update file only want sql
		}
		
		
		if (1)
		{
			while (my ($jj, $ww) = each(%alltabs))
			{
				$dumptabs_h{$jj} = $ww;
			}

			next # don't update file if dumping definitions
				if ($glob_glob->{dumpdef});
		}

		# backup the original version
		my $cpstat = `cp $filnam $backup_dir`;

		# write the modified file to the temp directory
		open my $file_out, "> " . File::Spec->catfile($glob_tmpdir, $basename)
			or die "Cannot open $basename for write in $glob_tmpdir";

		# write the modified file to the tmpdir
		print $file_out join("", @{$buffil});

		close $file_out;

	} # end for filnam

#	print "tmp: $tmp_indexing\tfil: $fil_indexing\n";

	# get catalog version if possible
	my $fil_catversion = File::Spec->catfile($dirs0, "catversion.h");
	my $catvstr = "";

	if (-e $fil_catversion)
	{
		$catvstr = `grep 'CATALOG_VERSION_NO' $fil_catversion`;

		if (length($catvstr))
		{
			my @cv = ($catvstr =~ m/(\d+)/im);
			$catvstr = "";
			$catvstr = $cv[0] if (scalar(@cv));
		}
	}

	if ($glob_glob->{syscache})
	{
		fixup_syscache(\%dumptabs_h, 	
						$fil_syscacheh, $tmp_syscacheh, 					
						$fil_syscachec, $tmp_syscachec, 					
						$verzion, $nnow);
	}

	if ($glob_glob->{sqldef})
	{
		open my $file_out, "> " . $glob_glob->{sqldef}
			or die "Cannot open $glob_glob->{sqldef} for write : $!";

		print $file_out 
			"-- Generated by " . $verzion . "\n-- on " . $nnow . "\n";

		if (length($catvstr))
		{
			print $file_out "-- CATALOG_VERSION_NO = " . $catvstr . "\n";
		}

		print $file_out "\n";
		print $file_out join(";\n", @arr_sql), ";\n";

		close $file_out;
		exit();
	}
	if ($glob_glob->{dumpdef})
	{
		open my $file_out, "> " . $glob_glob->{dumpdef}
			or die "Cannot open $glob_glob->{dumpdef} for write : $!";

		local $Data::Dumper::Sortkeys = 1;

        # JSON might not be installed, so test for it.

		if ($glob_glob->{dumpformat} =~ m/j/)
		{
			if (eval "require JSON")
			{
				# because JSON is REQUIREd, not USEd, the symbols are not
				# imported into the environment.

				# add a comment and sort the keys
				my $cmt = "Generated by $verzion on $nnow";
				$cmt .= " CATALOG_VERSION_NO=$catvstr"
					if (length($catvstr));
				$dumptabs_h{"__comment"} = $cmt;
				$dumptabs_h{"__info"} = {CATALOG_VERSION_NO => $catvstr}
					if (length($catvstr));

				my $whole_jfil = JSON::to_json(\%dumptabs_h, 
											   {pretty => 1, indent => 2, 
												canonical => 1});

				# remove date strings from relid_comment_tags to
				# prevent diffs and assuage Caleb's discomfiture
				#
				# s|20yymmdd */| */| 
				#
				$whole_jfil =~ s|\s+20(\d){6}\s+\*\/| \*/|gm;

				print $file_out $whole_jfil;
			}
			else
			{
				die("Fatal Error: The required package JSON is not installed -- please download it from www.cpan.org\n");
			}
		}
		else
		{
			print $file_out Data::Dumper->Dump([\%dumptabs_h]),"\n";
		}

		close $file_out;
		exit();
	}

	fixup_generated_code (\%alltabs, 
						  "indexing.h", $fil_indexing, $tmp_indexing,
		\@arr_indexing, $verzion, $nnow)
		if (scalar(@arr_indexing));

	fixup_generated_code (\%alltabs, 
						  "toasting.h", $fil_toastingh, $tmp_toastingh,
		\@arr_toastingh, $verzion, $nnow)
		if (scalar(@arr_toastingh));
	fixup_generated_code (\%alltabs, 
						  "toasting.c", $fil_toastingc, $tmp_toastingc,
		\@arr_toastingc, $verzion, $nnow)
		if (scalar(@arr_toastingc));

	fixup_generated_code (\%alltabs, 
						  "pg_type.h", $fil_pg_type, $tmp_pg_type,
		\@arr_pg_type, $verzion, $nnow)
		if (scalar(@arr_pg_type));

#	print Data::Dumper->Dump(\@arr_catalogc_tab), "\n";

	# do toast entries first because they are last (keep line
	# numbering consistent for prior items), then index, then table
	#
	# catalog.c IsSharedRelation index entries
	my ($catc_ref, $catc_ref2);

	$catc_ref2 = 
		fixup_generated_code (\%alltabs, 
							  "catalog.c toast", $fil_catalogc, $tmp_catalogc,
							  \@arr_catalogc_toast, $verzion, $nnow)
		if (scalar(@arr_catalogc_toast));

	if (scalar(@arr_catalogc_ind))
	{
		$catc_ref = 
		fixup_generated_code (\%alltabs, 
						  "catalog.c ind", $fil_catalogc, $tmp_catalogc,
							  \@arr_catalogc_ind, $verzion, $nnow, $catc_ref2);
	}
	else
	{
		$catc_ref = $catc_ref2;
	}

	# catalog.c IsSharedRelation table entries
	fixup_generated_code (\%alltabs, 
						  "catalog.c tab", $fil_catalogc, $tmp_catalogc,
						  \@arr_catalogc_tab, $verzion, $nnow, $catc_ref)
		if (scalar(@arr_catalogc_tab));

	# bootparse.y static type entries
	fixup_generated_code (\%alltabs, 
						  "bootparse.y", $fil_bootparse, $tmp_bootparse,
						  \@arr_bootparse, $verzion, $nnow)
		if (scalar(@arr_bootparse));
	
	if (scalar(keys(%h_pg_tidy)))
	{
		fixup_pg_tidy (\%h_pg_tidy, $fil_pg_tidy, $tmp_pg_tidy,
			$verzion, $nnow);
		fixup_makefile (\%h_pg_tidy, $fil_catmakem, $tmp_catmakem,
			$verzion, $nnow);
	}

}

exit();


# SLZY_TOP_BEGIN
if (0)
{
    my $bigstr = <<'EOF_bigstr';
{
   "args" : [
      {
         "alias" : "?",
         "long" : "Print a brief help message and exits.",
         "name" : "help",
         "required" : "0",
         "short" : "brief help message",
         "type" : "untyped"
      },
      {
         "long" : "Prints the manual page and exits.",
         "name" : "man",
         "required" : "0",
         "short" : "full documentation",
         "type" : "untyped"
      },
      {
         "alias" : "dd",
         "long" : "$ddlong",
         "name" : "dumpdef",
         "short" : "output file for dump of serialized catalog data structures",
         "type" : "outfile"
      },
      {
         "alias" : "df|dumpfmt",
         "long" : "Specify a format for the dumpfile.  The only valid options are jason or perl.",
         "name" : "dumpformat",
         "short" : "format options for dump file [perl, jason]",
         "type" : "outfile"
      },
      {
         "long" : "$sqldeflong",
         "name" : "sqldef",
         "short" : "output file for dump of catalog DDL statements",
         "type" : "outfile"
      },
      {
         "long" : "$syscachelong",
         "name" : "syscache",
         "short" : "build syscache entries",
         "type" : "untyped"
      }
   ],
   "long" : "$toplong",
   "properties" : {
      "COPYDATES" : "2009-2012",
      "slzy_date" : 1350599086
   },
   "short" : "generate catalog entries",
   "version" : "34"
}

EOF_bigstr
}
# SLZY_TOP_END

# SLZY_LONG_BEGIN
if (0)
{
	my $ddlong = <<'EOF_ddlong';
    
Specify an optional filename to hold a dump of the serialized catalog
data structures.  The format of the dump file is determined by
dumpformat
EOF_ddlong

my $syscachelong = <<'EOF_syscachelong';

If specified, rebuild syscache.h and syscache.c.  Note that this
option, like dumpdef, must read all catalog headers, ie in
src/include/catalog, the command:

   perl tidycat.pl -syscache *.h

constructs new versions of syscache.c and syscache.h.  

NOTE: Modification and extension of syscache entries is extremely rare.  
Usage of this option is discouraged.


EOF_syscachelong

my $sqldeflong = <<'EOF_sqldeflong';
    
Specify an optional filename to hold the CREATE TABLE statements from
the tidycat definitions.  Note that these statements will contain the
tidycat WITH clause, which is not valid SQL.
EOF_sqldeflong

	my $toplong = <<'EOF_longstr';
tidycat.pl handles all of your stinky catalog problems, leaving a
fresh, clean scent.  Catalog tables require several sets of
co-ordinated modifications to multiple source files to define the
table and indexes, and (under some circumstances) the toast tables and
indexes (in toasting.h and toasting.c), as well as some special code
in catalog.c and bootparse.y to aid in bootstrap and upgrade.  tidycat
also updates a generated list of headers in pg_tidycat.h and the
catalog Makefile.  The original files are copied to a special
tidycat_backup directory in /tmp, and all generated files are written
to /tmp.  tidycat.pl uses a single definition statement to generate
the code associated with the table in multiple source files.  A sample
definition for the fictional pg_foobar.h follows:

  /* TIDYCAT_BEGINDEF

   CREATE TABLE pg_foobar
   with (camelcase=FooBar, shared=true, oid=true, relid=9991, reltype_oid=9992)
   (
   fooname name, -- name of foo bar
   foolimit real, -- max active count limit
   fooignore boolean, -- ignore foo in baz context
   );

   create unique index on pg_foobar(oid) with (indexid=9993);
   create index on pg_foobar(fooname) with (indexid=9994);

   TIDYCAT_ENDDEF
  */

The definition must begin and end with the
TIDYCAT_BEGINDEF/TIDYCAT_ENDDEF exactly as shown.  The CREATE TABLE
statement is almost identical to standard SQL, with the addition of a
special WITH clause for implementation-specific features of the
catalog entry.  Currently, the relid and reltype_oid must be specified
using unassigned oids from the unused_oids script.  The options are:

{PODOVER8}

{ITEM} CamelCase: (optional)

If your tablename is a compound name, the index definitions look a
little nicer if you define an appropriate camelcase name.  Otherwise,
the default version of the name is the tablename, minus the "pg_" prefix, 
initial letter capitalized.

{ITEM} shared: (false by default)

Whether the table is local to each database or shared by all.

{ITEM} oid: (true by default)

Whether the table has an auto-generated oid column.

{ITEM} relid: (required)

The relid of the table in pg_class.  Use unused_oids to find one.

{ITEM} reltype_oid: (required for all post-3.3 tables)

The static reltype in pg_type (necessary for upgrade).  Use
unused_oids to find one.

{ITEM} toast_oid: (required for all tables with text or array columns)

The oid of the toast table (see toasting.h).  Use unused_oids to find
one.  tidycat will automatically detect if the table definition
requires a toast table and return an error if it is not specified.

{ITEM} toast_index: (required for all tables with toast_oid)

The oid of the index of the toast table (see toasting.h).  Use
unused_oids to find one.

{ITEM} toast_reltype: (required for all toast tables post-3.3)

The static reltype of the toast table in pg_type (necessary for
upgrade). Use unused_oids to find one.

{ITEM} content: (optional)

The "content" is only for catalog tables with non-standard content
management.  "Normal" catalog tables are replicated from the master to
all the segments.  Non-standard tables fall into three categories:
MASTER_ONLY, SEGMENT_LOCAL, and PERSISTENT. Don't add any new
non-standard tables.  Please.  Note that this flag controls the
generation of validation logic for checkcat; it does not control the
catalog table tuple replication mechanisms.

{PODBACK}

Similarly, index definitions are unique or non-unique, and require an
indexid (and an optional indexname).

Running tidycat.pl against pg_foobar.h adds the following section
after the definition:

  /* TIDYCAT_BEGIN_CODEGEN 
  
     WARNING: DO NOT MODIFY THE FOLLOWING SECTION: 
     Generated by tidycat.pl version 3.
     on Tue Dec  8 12:50:21 2009
  */
  
  
  /*
   TidyCat Comments for pg_foobar:
    Table is shared, so catalog.c:IsSharedRelation is updated.
    Table has an Oid column.
    Table has static type (see pg_types.h).
  
  */
  
  /* ----------------
   *		pg_foobar definition.  cpp turns this into
   *		typedef struct FormData_pg_foobar
   * ----------------
   */
  #define FooBarRelationId	9991
  
  CATALOG(pg_foobar,9991) BKI_SHARED_RELATION
  {
  	NameData	fooname;	/* name of foo bar */
  	float4		foolimit;	/* max active count limit */
  	bool		fooignore;	/* ignore foo in baz context */
  } FormData_pg_foobar;
  
  
  /* ----------------
   *		Form_pg_foobar corresponds to a pointer to a tuple with
   *		the format of pg_foobar relation.
   * ----------------
   */
  typedef FormData_pg_foobar *Form_pg_foobar;
  
  
  /* ----------------
   *		compiler constants for pg_foobar
   * ----------------
   */
  #define Natts_pg_foobar				3
  #define Anum_pg_foobar_fooname		1
  #define Anum_pg_foobar_foolimit		2
  #define Anum_pg_foobar_fooignore	3
  
  
  /* TIDYCAT_END_CODEGEN */
  

The generated code contains a CATALOG macro/struct definition for the
table, where the SQL datatypes are converted to C types.  The naming
and comments follow established conventions.

Additional modifications are made to indexing.h:

  /* relation id: 9991 - pg_foobar 20091208 */
  DECLARE_UNIQUE_INDEX(pg_foobar_oid_index, 9993, on pg_foobar using btree(oid oid_ops));
  #define FooBarOidIndexId 9993

  /* relation id: 9991 - pg_foobar 20091208 */
  DECLARE_INDEX(pg_foobar_fooname_index, 9994, on pg_foobar using btree(fooname name_ops));
  #define FooBarFoonameIndexId 9994


And the function IsSharedRelation() in catalog.c:
  bool 
  IsSharedRelation(Oid relationId)
  {
      /* These are the shared catalogs (look for BKI_SHARED_RELATION) */
      if (relationId == AuthIdRelationId ||
	(...much code...)
  /* relation id: 9991 - pg_foobar 20100105 */
  relationId == FooBarRelationId ||

Note that IsSharedRelation is only updated for shared tables.

And Boot_CreateStmt in bootparse.y:

  Boot_CreateStmt:
	  	  XCREATE optbootstrap optsharedrelation optwithoutoids boot_ident oidspec LPAREN
				{
	(...much code...)
  /* relation id: 9991 - pg_foobar 20100105 */
                            case FooBarRelationId:
                                typid = PG_FOOBAR_RELTYPE_OID;
                                break;

And pg_type.h:

  /* relation id: 9991 - pg_foobar 20100105 */
  DATA(insert OID = 9992 ( pg_foobar      PGNSP PGUID -1 f c t \054 9991 0 
	   record_in record_out record_recv record_send - d x f 0 -1 0 
       _null_ _null_ ));
  #define PG_FOOBAR_RELTYPE_OID 9992


{HEAD2} JSON document

In src/include/catalog, the command:

   perl tidycat.pl -dd foo.json -df json *.h

will generate a JSON document describing all of the catalog tables.
This file is installed under tools/bin/gppylib/data, and gpcheckcat
uses this data to generate check queries for foreign key constraint.

{HEAD1} CAVEATS

tidycat does not modify the original files -- it writes modified
versions of the files to /tmp.  You need to copy over the originals
with the generated files manually.  If you need to restore the
originals you can use the copies from the tidycat_backup directory.
Multiple cycles of tidycat with changing definitions can leave junk in
/tmp, and you might copy that junk into you source tree.  Do not copy
over any generated files that are older than your latest backup
directory.

EOF_longstr


}
# SLZY_LONG_END