<HTML>
<BODY>
Pig is a platform for a data flow programming on large data sets in a parallel
environment.  It consists of a language to specify these programs, 
<a href="http://wiki.apache.org/pig/PigLatin">Pig Latin</a>,
a compiler for this language, and an execution engine to execute the programs.
<p>
Pig runs on <a href="http://hadoop.apache.org/core/">hadoop</a>
MapReduce, reading data from and writing data to HDFS, and doing processing via
one or more MapReduce jobs.

<h2> Design </h2>
This section gives a very high overview of the design of the Pig system.  
Throughout the documents you can see design for that package or class by 
looking for the Design heading in the documentation.

<h3> Overview </h3>
<p>
Pig's design is guided by our <a href="http://hadoop.apache.org/pig/philosophy.html">
pig philosophy</a>.
<p>
Pig shares many similarities with a traditional RDBMS design.  It has a parser,
type checker, optimizer, and operators that perform the data processing.  However,
there are some 
significant differences.  Pig does not have a data catalog, there are no
transactions, pig does not directly manage data storage, nor does it implement the 
execution framework.
<p>
<h3> High Level Architecture </h3>
Pig is split between the front and back ends of the engine.  In the front end,
the parser transforms a Pig Latin script into a logical plan.
Semantic checks (such
as type checking) and some optimizations (such as determining which fields in the data need
to be read to satisfy the script) are done on this Logical Plan.  The Logical
Plan is than transformed into a
{@link org.apache.pig.backend.hadoop.executionengine.physicalLayer.plans.PhysicalPlan}.
This Physical Plan contains the operators that will be applied to the data.  This is then
divided into a set of MapReduce jobs by the
{@link org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler} into an
{@link org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.plans.MROperPlan}.  This
MROperPlan (aka the map reduce plan) is then optimized (for example, the combiner is used where
possible, jobs that scan the same input data are combined where possible, etc.).  Finally a set of
MapReduce jobs are generated by the
{@link org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler}.  These are
submitted to Hadoop and monitored by the
{@link org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher}.
<p>
On the backend, each 
{@link org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce.Map},
{@link org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner.Combine}, and 
{@link org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce.Reduce} 
use the pipeline of physical operators constructed in the front end to load, process, and store
data.

<h3> Programmatic Interface </h3>
<p>
In addition to the command line and grunt interfaces, users can connect to 
{@link org.apache.pig.PigServer} from a Java program.
<p>
Pig makes it easy for users to extend its functionality by implementing User Defined Functions
(UDFs).  There are interfaces for defining functions to load data
{@link org.apache.pig.LoadFunc}, storing data {@link org.apache.pig.StoreFunc}, doing evaluations
on fields (including collections of data, so user defined aggregates are possible) 
{@link org.apache.pig.EvalFunc} and filtering data {@link org.apache.pig.FilterFunc}.
</BODY>
</HTML>