| commit | 163fc26ad9abe30f3a9c0be4622c6e4e187e8505 | [log] [tgz] |
|---|---|---|
| author | Matthew Hayes <matthew.terence.hayes@gmail.com> | Sat May 23 13:00:44 2015 -0700 |
| committer | Matthew Hayes <mhayes@Matthews-MacBook-Pro-2.local> | Wed May 27 13:06:20 2015 -0700 |
| tree | 7c9c7e0bb3282b5a2721ef2f04684899542aa0c9 | |
| parent | 745a230f2ab5bdd3d6f2017a0b89edf30aebcce3 [diff] |
DATAFU-58 Hadoop 2.0 Upgrade dependencies to Hadoop 2.7.0 and Pig 0.14.0 and fix all testing issues
Apache DataFu is a collection of libraries for working with large-scale data in Hadoop. The project was inspired by the need for stable, well-tested libraries for data mining and statistics.
It consists of two libraries:
For more information please visit the website:
If you'd like to jump in and get started, check out the corresponding guides for each library:
Bugs and feature requests can be filed here. For other help please see the website.
To build DataFu from a git checkout or binary release, run:
./gradlew clean assemble
To build DataFu from a source release, it is first necessary to download the gradle wrapper script above. This bootstrapping process requires Gradle to be installed on the source machine. Gradle is available through most package managers or directly from its website. To bootstrap the wrapper, run:
gradle -b bootstrap.gradle
After the bootstrap script has completed, the regular gradlew instructions are available.
The datafu-pig JAR can be found under datafu-pig/build/libs by the name datafu-pig-x.y.z.jar, where x.y.z is the version. Similarly, the datafu-hourglass can be found in the datafu-hourglass/build/libs directory.
This command generates the eclipse project and classpath files:
./gradlew eclipse
To clean up the eclipse files:
./gradlew cleanEclipse
To run all the tests:
./gradlew test
To run only the DataFu Pig tests:
./gradlew :datafu-pig:test
To run only the DataFu Hourglass tests:
./gradlew :datafu-hourglass:test
To run tests for a single class, use the test.single property. For example, to run only the QuantileTests:
./gradlew :datafu-pig:test -Dtest.single=QuantileTests
The tests can also be run from within eclipse. You'll need to install the TestNG plugin for Eclipse. See: http://testng.org/doc/download.html.
Potential issues and workaround: