Google Git
Sign in
apache / datafu / ad4766482e21638e3d62977d05db711d72e453e5
commitad4766482e21638e3d62977d05db711d72e453e5[log] [tgz]
authorMatthew Hayes <matthew.terence.hayes@gmail.com>Tue Nov 25 22:20:25 2014 -0800
committerMatthew Hayes <matthew.terence.hayes@gmail.com>Thu Nov 27 07:48:24 2014 -0800
treecaab2e4aa8a950e658ff6b1821d1ba004ae087e2
parent1563535f20cba433497664262717d861775c5f9b [diff]
DATAFU-76 Resolve issues with build-plugin directory

https://issues.apache.org/jira/browse/DATAFU-76

I've attempted to clean up the NOTICE and LICENSE files.  For reference I read http://www.apache.org/dev/licensing-howto.html.  I removed everything form the NOTICE because I don't think they were actually required to be there.  Many are not actually bundled, and for the ones that are they do not have a NOTICE, so there is nothing that needs to be propagated in our NOTICE file.  The one exception is commons-math, which does have a NOTICE file.  I still need to update the NOTICE with those files that we do include from there.

I also updated the LICENSE file to include licensing info for files that we bundle.
3 files changed
tree: caab2e4aa8a950e658ff6b1821d1ba004ae087e2
  1. .settings/
  2. build-plugin/
  3. datafu-hourglass/
  4. datafu-pig/
  5. examples/
  6. gradle/
  7. site/
  8. .classpath.template
  9. .factorypath.template
  10. .gitignore
  11. .travis.yml
  12. bootstrap.gradle
  13. build.gradle
  14. changes.md
  15. CONTRIBUTORS
  16. doap_DataFu.rdf
  17. gradle.properties
  18. gradlew
  19. HEADER
  20. LICENSE
  21. NOTICE
  22. README.md
  23. RELEASE.md
  24. releasing.md
  25. settings.gradle
  26. settings.xml.template
  27. test.sh
  28. test_in_background.sh
README.md

Apache DataFu

Apache DataFu is a collection of libraries for working with large-scale data in Hadoop. The project was inspired by the need for stable, well-tested libraries for data mining and statistics.

It consists of two libraries:

  • Apache DataFu Pig: a collection of user-defined functions for Apache Pig
  • Apache DataFu Hourglass: an incremental processing framework for Apache Hadoop in MapReduce

For more information please visit the website:

If you'd like to jump in and get started, check out the corresponding guides for each library:

Blog Posts

Presentations

Papers

Getting Help

Bugs and feature requests can be filed here. For other help please see the website.

Developers

Building the Code

To build DataFu from a git checkout or binary release, run:

./gradlew clean assemble

To build DataFu from a source release, it is first necessary to download the gradle wrapper script above. This bootstrapping process requires Gradle to be installed on the source machine. Gradle is available through most package managers or directly from its website. To bootstrap the wrapper, run:

gradle -b bootstrap.gradle

After the bootstrap script has completed, the regular gradlew instructions are available.

The datafu-pig JAR can be found under datafu-pig/build/libs by the name datafu-pig-x.y.z.jar, where x.y.z is the version. Similarly, the datafu-hourglass can be found in the datafu-hourglass/build/libs directory.

Generating Eclipse Files

This command generates the eclipse project and classpath files:

./gradlew eclipse

To clean up the eclipse files:

./gradlew cleanEclipse

Running the Tests

To run all the tests:

./gradlew test

To run only the DataFu Pig tests:

./gradlew :datafu-pig:test

To run only the DataFu Hourglass tests:

./gradlew :datafu-hourglass:test

To run tests for a single class, use the test.single property. For example, to run only the QuantileTests:

./gradlew :datafu-pig:test -Dtest.single=QuantileTests

The tests can also be run from within eclipse. Note that you may run out of heap when executing tests in Eclipse. To fix this adjust your heap settings for the TestNG plugin. Go to Eclipse->Preferences. Select TestNG->Run/Debug. Add “-Xmx1G” to the JVM args.