Google Git
Sign in
apache / datafu / f75f9c4fdf9b94944b3d51cf6dab256e077899e7
commitf75f9c4fdf9b94944b3d51cf6dab256e077899e7[log] [tgz]
authorPhilip (flip) Kromer <flip@infochimps.org>Mon May 19 09:12:34 2014 -0500
committerMatthew Hayes <matthew.terence.hayes@gmail.com>Tue May 20 07:27:18 2014 -0700
tree1dcb6a1f27c56e0628a70225f7249dc5ba8e7edc
parent3cfcea78d5dfed961625f8dfce0f75579ac0b0ff [diff]
DATAFU-49 Examples work again

* Removed dependency on Guava (which it didn't actually depend on) and on Piggybank (date functions are now first class)
* Path to datafu jar correct for current repo layout
* I made the quantile examples demonstrate a comparison of the exact vs approx algorithms
* Added the script to generate data for quantile.
* Quantile examples demonstrate both ways of constructing a Quantile UDF (number of partitions vs list of breakpoints)

https://issues.apache.org/jira/browse/DATAFU-49

Signed-off-by: Matthew Hayes <matthew.terence.hayes@gmail.com>
7 files changed
tree: 1dcb6a1f27c56e0628a70225f7249dc5ba8e7edc
  1. .settings/
  2. build-plugin/
  3. cobertura/
  4. datafu-hourglass/
  5. datafu-pig/
  6. eclipselibs/
  7. examples/
  8. gradle/
  9. licenses/
  10. site/
  11. staticlibs/
  12. tools/
  13. .classpath.template
  14. .factorypath.template
  15. .gitignore
  16. .travis.yml
  17. build.gradle
  18. changes.md
  19. CONTRIBUTORS
  20. gradle.properties
  21. gradlew
  22. HEADER
  23. LICENSE
  24. NOTICE
  25. README.md
  26. releasing.md
  27. settings.gradle
  28. settings.xml.template
  29. test.sh
  30. test_in_background.sh
README.md

Apache DataFu

Apache DataFu is a collection of libraries for working with large-scale data in Hadoop. The project was inspired by the need for stable, well-tested libraries for data mining and statistics.

It consists of two libraries:

  • Apache DataFu Pig: a collection of user-defined functions for Apache Pig
  • Apache DataFu Hourglass: an incremental processing framework for Apache Hadoop in MapReduce

For more information please visit the website:

If you'd like to jump in and get started, check out the corresponding guides for each library:

Blog Posts

Presentations

Papers

Getting Help

Bugs and feature requests can be filed here. For other help please see the discussion group.

Developers

DataFu Pig

Building the Code

The Apache DataFu Pig library can be built by running the command below. More information about working with the source code can be found in the DataFu Pig Contributing Guide.

./gradlew assemble

The built JAR can be found under datafu-pig/build/libs by the name datafu-pig-x.y.z.jar, where x.y.z is the version.

Generating Eclipse Files

This command generates the eclipse project and classpath files:

./gradlew eclipse

To clean up the eclipse files:

./gradlew cleanEclipse

Running the Tests

To run all the tests:

./gradlew test

To run tests for a single class, use the test.single property. For example, to run only the QuantileTests:

/gradlew :datafu-pig:test -Dtest.single=QuantileTests

The tests can also be run from within eclipse.

DataFu Hourglass

Building the Code

The Apache DataFu Pig library can be built by running the commands below. More information about working with the source code can be found in the DataFu Hourglass Contributing Guide.

cd contrib/hourglass
ant jar