Google Git
Sign in
apache / datafu / e59348fc687a3492e9e0d75f22e1f6ee1d92fa06
commite59348fc687a3492e9e0d75f22e1f6ee1d92fa06[log] [tgz]
authorMatthew Hayes <matthew.terence.hayes@gmail.com>Fri Jul 04 09:37:57 2014 -0700
committerMatthew Hayes <matthew.terence.hayes@gmail.com>Fri Jul 04 09:38:01 2014 -0700
treeab3119cef6771b278ce54018d6f2e7fcc2983210
parent412b14bc9169a8f4f7433f02551cbbf698846f20 [diff]
DATAFU-55 Remove legacy checked in jars

These jars are no longer used.  They are remnants of the previous Ant build system.

Includes: cobertura, autojar, jarjar, duplicate eclipse libs
31 files changed
tree: ab3119cef6771b278ce54018d6f2e7fcc2983210
  1. .settings/
  2. build-plugin/
  3. datafu-hourglass/
  4. datafu-pig/
  5. examples/
  6. gradle/
  7. licenses/
  8. site/
  9. staticlibs/
  10. .classpath.template
  11. .factorypath.template
  12. .gitignore
  13. .travis.yml
  14. build.gradle
  15. changes.md
  16. CONTRIBUTORS
  17. gradle.properties
  18. gradlew
  19. HEADER
  20. LICENSE
  21. NOTICE
  22. README.md
  23. releasing.md
  24. settings.gradle
  25. settings.xml.template
  26. test.sh
  27. test_in_background.sh
README.md

Apache DataFu

Apache DataFu is a collection of libraries for working with large-scale data in Hadoop. The project was inspired by the need for stable, well-tested libraries for data mining and statistics.

It consists of two libraries:

  • Apache DataFu Pig: a collection of user-defined functions for Apache Pig
  • Apache DataFu Hourglass: an incremental processing framework for Apache Hadoop in MapReduce

For more information please visit the website:

If you'd like to jump in and get started, check out the corresponding guides for each library:

Blog Posts

Presentations

Papers

Getting Help

Bugs and feature requests can be filed here. For other help please see the discussion group.

Developers

DataFu Pig

Building the Code

The Apache DataFu Pig library can be built by running the command below. More information about working with the source code can be found in the DataFu Pig Contributing Guide.

./gradlew assemble

The built JAR can be found under datafu-pig/build/libs by the name datafu-pig-x.y.z.jar, where x.y.z is the version.

Generating Eclipse Files

This command generates the eclipse project and classpath files:

./gradlew eclipse

To clean up the eclipse files:

./gradlew cleanEclipse

Running the Tests

To run all the tests:

./gradlew test

To run tests for a single class, use the test.single property. For example, to run only the QuantileTests:

./gradlew :datafu-pig:test -Dtest.single=QuantileTests

The tests can also be run from within eclipse.

DataFu Hourglass

Building the Code

The Apache DataFu Pig library can be built by running the commands below. More information about working with the source code can be found in the DataFu Hourglass Contributing Guide.

cd contrib/hourglass
ant jar