Pig Change Log

Trunk (unreleased changes)

  INCOMPATIBLE CHANGES

  IMPROVEMENTS 

  OPTIMIZATIONS

  BUG FIXES

Release 0.1.1 - Unreleased

INCOMPATIBLE CHANGES

NEW FEATURES

IMPROVEMENTS

PIG-253: integration with hadoop-18

BUG FIXES

PIG-342: Fix DistinctDataBag to recalculate size after it has spilled. (bdimcheff via gates)

Release 0.1.0 - - 2008-09-11

  INCOMPATIBLE CHANGES

  PIG-123: requires escape of '\' in chars and string

  NEW FEATURES

  PIG-20 Added custom comparator functions for order by (phunt via gates)
  PIG-94: Streaming implementation
  PIG-58: parameter substitution
  PIG-55: added custom splitter (groves via olgan)
  PIG-59: Add a new ILLUSTRATE command (shubhamc via gates).
  PIG-256: Added variable argument support for UDFs (pi_song)

  IMPROVEMENTS 

    PIG-8 added binary comparator (olgan)

    PIG-11 Add capability to search for jar file to register. (antmagna via
    olgan)
  
    PIG-7: Added use of combiner in some restricted cases. (gates)

    PIG-47: Added methods to DataMap to provide access to its content

    PIG-30: Rewrote DataBags to better handle decisions of when to spill to
    disk and to spill more intelligently. (gates)

    PIG-12: Added time stamps to log4j messages (phunt via gates).

    PIG-44: Added adaptive decision of the number of records to hold in memory
    before spilling (utkarsh)

    PIG-56: Made DataBag implement Iterable. (groves via gates)

    PIG-39: created more efficient version of read (spullara via olgan)

    PIG-32: ABstraction layer (olgan)

    PIG-83: Change everything except grunt and Main (PigServer on down) to use
    common logging abstraction instead of log4j.  By default in grunt, log4j
    still used as logging layer.  Also converted all System.out/err.println
    statements to use logging instead. (francisoud via gates)

    PIG-13: adding version to the system (joa23 via olgan)

    PIG-113:  Make explain output more understandable (pi_song via gates)

    PIG-120:  Support map reduce in local mode.  To do this user needs to
    specify execution type as mapreduce and cluster name as local (joa23 via
    gates).

    PIG-106:  Change StringBuffer and String '+' to StringBuilder (francisoud
    via gates).

    PIG-111: Reworked configuration to be setable via properties.
    (Contributions from joa23, pi_song, and oae via gates).

  BUG FIXES

    PIG-24 Files that were incorrectly placed under test/reports have been
    removed.  ant clean now cleans test/reports.  (milindb via gates)

    PIG-25 com.yahoo.pig dir left under pig/test by mistake. removed it (olgan@)

    PIG-23 Made pig work with java 1.5. (milindb via gates)

    PIG-17 integrated with Hadoop 0.15 (olgan@)

    PIG-33 Help was commented out - uncommented (olgan)

    PIG-31: second half of concurrent mode problem addressed (olgan)

    PIG-14: added heartbeat functionality (olgan)

    PIG-17: updated hadoop15.jar to match hadoop 0.15.1 release

    PIG-29: fixed bag factory to be properly initialized (utkarsh)

    PIG-43: fixed problem where using the combiner prevented a pig alias
    from being evaluated more than once. (gates)

    PIG-45: Fixed pig.pl to not assume hodrc file is named the same as
    cluster name (gates).

    PIG-7 (more): Fixed bug in PigCombiner where it was writing IndexedTuples
    instead of Tuples, causing Reducer to crash in some cases.

    PIG-41: Added patterns to svn:ignore

    PIG-51: Fixed combiner in the presence of flattening

    PIG-61: Fixed MapreducePlanCompiler to use PigContext to load up the
    comparator function instead of Class.forName.  (gates)

    PIG-63: Fix for non-ascii UTF-8 data (breed@ and olgan@)

    PIG-77: Added eclipse specific files to svn:ignore

    PIG-57: Fixed NPE in PigContext.fixUpDomain (francisoud via gates)

    PIG-69: NPE in PigContext.setJobtrackerLocation (francisoud via gates)

    PIG-78: src/org/apache/pig/builtin/PigStorage.java doesn't compile (arun
    via olgan)

    PIG-87: Fix pig.pl to find java via JAVA_HOME instead of hardcoded default
    path.  Also fix it to not die if pigclient.conf is missing. (craigm via
    gates).

    PIG-89: Fix DefaultDataBag, DistinctDataBag, SortedDataBag to close spill
    files when they are done spilling (contributions by craigm, breed, and
    gates, committed by gates).

    PIG-95: Remove System.exit() statements from inside pig (joa23 via gates).

    PIG-65: convert tabs to spaces (groves via olgan)

    PIG-97: Turn off combiner in the case of Cogroup, as it doesn't work when
    more than one bag is involved (gates).

    PIG-92: Fix NullPointerException in PIgContext due to uninitialized conf
    reference. (francisoud via gates)

    PIG-80: In a number of places stack trace information was being lost by an
    exception being caught, and a different exception then thrown.  All those
    locations have been changed so that the new exception now wraps the old.
    (francisoud via gates).

    PIG-84: Converted printStackTrace calls to calls to the logger.
    (francisoud via gates).

    PIG-88: Remove unused HadoopExe import from Main.  (pi_song via gates).

    PIG-99: Fix to make unit tests not run out of memory. (francisoud via
    gates).

    PIG-107: enabled several tests. (francisoud via olgan)

    PIG-46: abort processing on error for non-interactive mode (olston via
    olgan)

    PIG-109: improved exception handling (oae via olgan)

    PIG-72: Move unit tests to use MiniDFS and MiniMR so that unit tests can
    be run w/o access to a hadoop cluster. (xuzh via gates)

    PIG-68: improvements to build.xml (joa23 via olgan)

    PIG-110: Replaced code accidently merged out in PIG-32 fix that handled
    flattening the combiner case. (gates and oae)

    PIG-68 broke the build process by hardwiring hadoop15 jar for the purpose
    of compile. Fixed that (olgan)
    
    PIG-124: only run one test (ant test -Dtestcase=TestMapReduce) not the 
    complete test suite (xuzh vi olgan)

    PIG-127: changes to build.xml to have description for each target
    (francisoud via olgan)

    PIG-101: changes in tests to use enum type (francisoud via olgan)

    PIG-125: Improve exception handling in cases when an attempt is made to
    access a field as a tuple, and it turns out not to be a tuple (oae via
    gates).

    PIG-13: make the code use svn only if available (joa23 via olgan)

    PIG-118: make sure union/join/cross takes 2 params (pi_song vi olgan)

    PIG-94: M1 for streaming: maps and reduce side support with default
    (de)serializer (acmurthy via olgan)

    PIG-129: making sure that temp files are stored in task's home dir and
    cleaned up

    PIG-115: Removed Yahoo specific scripts/pig.pl, replaced with generic
    bash script bin/pig.  Moved startHOD.expect to bin (joa23 via gates).

    PIG-18: changes to make pig work with Hadoop 0.16 and HOD 0.4 (olgan)

    PIG-164:  Fix memory issue in SpillableMemoryManager to partially clean the list of
    bags each time a new bag is added rather than waiting until the garbage
    collector tells us we are out of memory (gates).

    PIG-154: moving parsing for DEFINE and STORE into QueryParser

    PIG-100: improved error handling

    PIG-94: changes for M2 of streaming: input/ouptut/ ship/cache error
    handling

    PIG-108: Fixed PigCombine to not do initialization on every call to
    reduce, but instead only do it once in the call to configure.  (joa23 via
    gates).

    PIG-172: dealing with NULL error messages in exceptions (olgan)

    PIG-170: sort bags so that largest ones are released first

    PIG-122: Added build and src-gen to the list of ignore files in
    the top level directory (joa23 via gates).

    PIG-94: M3 code update for streaming (arunc via olgan)

    PIG-179: Changed PigRecordReader to be a static singleton rather than
    thread local.  (gates).
    
    PIG-174,180: bug fixes in streaming (arunc via olgan)

    PIG-181: streaming bug fixing (arunc via olgan)

    PIG-182: streaming bug fix (arunc via olgan)

    PIG-184: streaming bug fixes

    PIG-153: Incorrect result caused by dump in between statements (pi_song
    via gates).

    PIG-178: Use of schema on secondary output of SPLIT throws
    IndexOutOfBoundsException (kali via gates).

    PIG-203: Fix bug in parameter substitution code where any pig script over
    1k caused pig to freeze. (kali via gates)

    PIG-204: Repair broken input splits (acmurthy via gates).

    PIG-188: Fix mismatches between pig slicer changes and new streaming
    feature (acmurthy via gates).

    PIG-149, PIG-150:  Fix doc target so that ant can generate docs (xuzh via
    gates).

    PIG-183:  Catch when a UDF has been compiled with the wrong version of
    java and give a RuntimeException (pi_song via gates).

    PIG-114: store one alias/logicalPlan twice leads to instantiation of
    StoreFunc as LoadFunc (pi_song via gates).

    PIG-213: Remove non-static references to logger from data bags and tuples,
    as it causes significant overhead (vgeschel via gates).

    PIG-216: Fix streaming to work with commands that use unix pipes (acmurthy
    via gates).

    PIG-207: Fix illustrate command to work in mapreduce mode (shubhamc via
    gates).

    PIG-218: Fixed param generation to work with arbitrary commands

    PIG-220: Fixed definition of parameter name for param substitution

    PIG-151: fixes to code that handles bzip files

    PIG-222: fix build break

    PIG-226: fix for streaming optimization bug (acmurthy via olgan)

    PIG-228: make multiple streaming outputs adhere to spec (acmurthy via olgan)

    PIG-224: fix to error handling code to produce correct error code

    PIG-176: Change bag spilling so that bags below a certain threshold are
    not spilled, thus avoiding proliferation of small files (pi_song via
    gates).

    PIG-227: making load/store function optional in stream input/output spec
    (acmurthy via olgan)

    PIG-215: Cleanup a few dangling ends left by PIG-111 (pi_song via gates).
    
    PIG-229: Proper error handling in case of deserializer failure

    PIG-230: Handling shipment for multiple ship/cache commands (acmurthy via
    olgan)

    PIG-219: Change unit tests to run both local and map reduce modes
    (kali via gates).

    PIG-202: Fix Order by so that user provided comparator func is used for
    quantile determination (kali via gates).

    PIG-231: validation for ship, cache, and skippath  (acmurthy via olgan)

    PIG-232: fix for number of output records when BinaryStirage is used
    (acmurthy via olgan)

    PIG-232: fix for number of input records when BinaryStirage is used
    (acmurthy via olgan)

    PIG-232: let valid cache specifications through (acmurthy via olgan)

    PIG-237: validation of the output directory (pi_song via olgan)

    PIG-236: Fix properties so that values specified via the
    command line (-D) are not ignored (pkamath via gates).

    PIG-198: integration with hadoop 17 (acmurthy via olgan)

    PIG-85: allowing control characters as delimiters for PigStorage (pi_song
    via olgan)

    PIG-250: disabling speculative execution (olgan)

    PIG-250: re-enabling speculative execution and fixing the failure
    (acmurthy via olgan)

    PIG-85: memory optimization (pi_song via olgan)

    PIG-243: Fixing unit tests on windows (daijy via olgan)

    PIG-198: Fixed pig script to pick up hadoop 17 instead of 15 (pi_song via gates). 

    PIG-266: fix warnings caused by HOD (olgan)

    PIG-245: added math functions to the piggybank (ajaygarg via olgan)

    PIG-255: make non-default constructors work for algebraic functions
    (ajaygarg via olgan)

    PIG-272: problem with streaming and intermediate store (acmurthy via olgan)

    PIG-243: make unit tests work on windows (daijy via olgan)

    PIG-271: added tutorial to SVN (olgan)

    PIG-235: memory management improvements (pkamath via olgan)

    PIG-284: target for building source jar (oae via olgan)

    PIG-34: added missing licenses

    PIG-34: added LICENSE, NOTICE and README file

    PIG-291: hod.param parameters not passed properly (thatha via olgan)

    PIG-34: changes to build process to create distribution tar file

    PIG-34: updated CHANGES.txt

	PIG-472: Added RegExLoader to piggybank, an abstract loader class to parse
	text files via regular espressions (spackest via gates)

	PIG-473: Added CommonLogLoader, a subclass of RegExLoader to piggybank (spackest via gates)

	PIG-474: Added MyRegexLoader, a subclass of RegExLoader, to piggybank (spackest via gates)