| {"name":"DataFu","body":"# DataFu\r\n\r\nDataFu is a collection of user-defined functions for working with large-scale data in Hadoop and Pig. This library was born out of the need for a stable, well-tested library of UDFs for data mining and statistics. It is used at LinkedIn in many of our off-line workflows for data derived products like \"People You May Know\" and \"Skills\". It contains functions for:\r\n\r\n* PageRank\r\n* Quantiles (median), variance, etc.\r\n* Sessionization\r\n* Convenience bag functions (e.g., set operations, enumerating bags, etc)\r\n* Convenience utility functions (e.g., assertions, easier writing of\r\nEvalFuncs)\r\n* and [more](http://sna-projects.com/datafu/javadoc/0.0.4/)...\r\n\r\nEach function is unit tested and code coverage is being tracked for the entire library. It has been tested against pig 0.9.\r\n\r\n[http://sna-projects.com/datafu/](http://sna-projects.com/datafu/)\r\n\r\n## What can you do with it?\r\n\r\nHere's a taste of what you can do in Pig.\r\n\r\n### Statistics\r\n \r\nCompute the [median](http://en.wikipedia.org/wiki/Median) of sequence of sorted bags:\r\n\r\n define Median datafu.pig.stats.Median();\r\n\r\n -- input: 3,5,4,1,2\r\n input = LOAD 'input' AS (val:int);\r\n\r\n grouped = GROUP input ALL;\r\n\r\n -- produces median of 3\r\n medians = FOREACH grouped {\r\n sorted = ORDER input BY val;\r\n GENERATE Median(sorted.val);\r\n }\r\n \r\nSimilarly, compute any arbitrary [quantiles](http://en.wikipedia.org/wiki/Quantile):\r\n\r\n define Quantile datafu.pig.stats.Quantile('0.0','0.5','1.0');\r\n\r\n -- input: 9,10,2,3,5,8,1,4,6,7\r\n input = LOAD 'input' AS (val:int);\r\n\r\n grouped = GROUP input ALL;\r\n\r\n -- produces: (1,5.5,10)\r\n quantiles = FOREACH grouped {\r\n sorted = ORDER input BY val;\r\n GENERATE Quantile(sorted.val);\r\n }\r\n\r\n### Set Operations\r\n\r\nTreat sorted bags as sets and compute their intersection:\r\n\r\n define SetIntersect datafu.pig.bags.sets.SetIntersect();\r\n \r\n -- ({(3),(4),(1),(2),(7),(5),(6)},{(0),(5),(10),(1),(4)})\r\n input = LOAD 'input' AS (B1:bag{T:tuple(val:int)},B2:bag{T:tuple(val:int)});\r\n\r\n -- ({(1),(4),(5)})\r\n intersected = FOREACH input {\r\n sorted_b1 = ORDER B1 by val;\r\n sorted_b2 = ORDER B2 by val;\r\n GENERATE SetIntersect(sorted_b1,sorted_b2);\r\n }\r\n \r\nCompute the set union:\r\n\r\n define SetUnion datafu.pig.bags.sets.SetUnion();\r\n\r\n -- ({(3),(4),(1),(2),(7),(5),(6)},{(0),(5),(10),(1),(4)})\r\n input = LOAD 'input' AS (B1:bag{T:tuple(val:int)},B2:bag{T:tuple(val:int)});\r\n\r\n -- ({(3),(4),(1),(2),(7),(5),(6),(0),(10)})\r\n unioned = FOREACH input GENERATE SetUnion(B1,B2);\r\n \r\nOperate on several bags even:\r\n\r\n intersected = FOREACH input GENERATE SetUnion(B1,B2,B3);\r\n\r\n### Bag operations\r\n\r\nConcatenate two or more bags:\r\n\r\n define BagConcat datafu.pig.bags.BagConcat();\r\n\r\n -- ({(1),(2),(3)},{(4),(5)},{(6),(7)})\r\n input = LOAD 'input' AS (B1: bag{T: tuple(v:INT)}, B2: bag{T: tuple(v:INT)}, B3: bag{T: tuple(v:INT)});\r\n\r\n -- ({(1),(2),(3),(4),(5),(6),(7)})\r\n output = FOREACH input GENERATE BagConcat(B1,B2,B3);\r\n\r\nAppend a tuple to a bag:\r\n\r\n define AppendToBag datafu.pig.bags.AppendToBag();\r\n\r\n -- ({(1),(2),(3)},(4))\r\n input = LOAD 'input' AS (B: bag{T: tuple(v:INT)}, T: tuple(v:INT));\r\n\r\n -- ({(1),(2),(3),(4)})\r\n output = FOREACH input GENERATE AppendToBag(B,T);\r\n\r\n### PageRank\r\n\r\nRun PageRank on a large number of independent graphs:\r\n\r\n define PageRank datafu.pig.linkanalysis.PageRank('dangling_nodes','true');\r\n\r\n topic_edges = LOAD 'input_edges' as (topic:INT,source:INT,dest:INT,weight:DOUBLE);\r\n\r\n topic_edges_grouped = GROUP topic_edges by (topic, source) ;\r\n topic_edges_grouped = FOREACH topic_edges_grouped GENERATE\r\n group.topic as topic,\r\n group.source as source,\r\n topic_edges.(dest,weight) as edges;\r\n\r\n topic_edges_grouped_by_topic = GROUP topic_edges_grouped BY topic; \r\n\r\n topic_ranks = FOREACH topic_edges_grouped_by_topic GENERATE\r\n group as topic,\r\n FLATTEN(PageRank(topic_edges_grouped.(source,edges))) as (source,rank);\r\n\r\n skill_ranks = FOREACH skill_ranks GENERATE\r\n topic, source, rank;\r\n \r\nThis implementation stores the nodes and edges (mostly) in memory. It is therefore best suited when one needs to compute PageRank on many reasonably sized graphs in parallel.\r\n \r\n## Start Using It\r\n\r\nThe JAR can be found [here](http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22com.linkedin.datafu%22) in the Maven central repository. The GroupId and ArtifactId are `com.linkedin.datafu` and `datafu`, respectively.\r\n\r\nIf you are using Ivy:\r\n\r\n <dependency org=\"com.linkedin.datafu\" name=\"datafu\" rev=\"0.0.4\"/>\r\n \r\nIf you are using Maven:\r\n\r\n <dependency>\r\n <groupId>com.linkedin.datafu</groupId>\r\n <artifactId>datafu</artifactId>\r\n <version>0.0.4</version>\r\n </dependency>\r\n \r\nOr you can download one of the packages from the [downloads](https://github.com/linkedin/datafu/downloads) section. \r\n\r\n## Working with the source code\r\n\r\nHere are some common tasks when working with the source code.\r\n\r\n### Build the JAR\r\n\r\n ant jar\r\n \r\n### Run all tests\r\n\r\n ant test\r\n\r\n### Run specific tests\r\n\r\nOverride `testclasses.pattern`, which defaults to `**/*.class`. For example, to run all tests defined in `QuantileTests`:\r\n\r\n ant test -Dtestclasses.pattern=**/QuantileTests.class\r\n\r\n### Compute code coverage\r\n\r\n ant coverage\r\n\r\n## Contribute\r\n\r\nThe source code is available under the Apache 2.0 license. \r\n\r\nFor help please see the [discussion group](http://groups.google.com/group/datafu). Bugs and feature requests can be filed [here](http://linkedin.jira.com/browse/DATAFU).","tagline":"Hadoop library for large-scale data processing","google":"UA-30533336-1","note":"Don't delete this file! It's used internally to help with page regeneration."} |