Dag dag = new Dag(); dag.addNode(new Node("filterImps", 10, "")); dag.addNode(new Node("filterClicks", 5, "")); dag.addNode(new Node("impClickJoin", 10, "filterImps,filterClicks")); dag.addNode(new Node("impCountsByGender", 10, "filterImps")); dag.addNode(new Node("impCountsByCountry", 10, "filterImps")); dag.addNode(new Node("clickCountsByGender", 5, "impClickJoin")); dag.addNode(new Node("clickCountsByCountry", 5, "impClickJoin")); dag.addNode(new Node("report",1,"impCountsByGender,impCountsByCountry,clickCountsByGender,clickCountsByCountry"));
In order to run the demo, use the following steps
See http://redis.io/topics/quickstart on how to install redis server
Start redis e.g: ./redis-server --port 6379 git clone https://git-wip-us.apache.org/repos/asf/helix.git cd recipes/task-execution mvn clean install package -DskipTests cd target/task-execution-pkg/bin chmod +x task-execution-demo.sh ./task-execution-demo.sh 2181 localhost 6379
+-----------------+ +----------------+ | filterImps | | filterClicks | | (parallelism=10)| | (parallelism=5)| +----------+-----++ +-------+--------+ | | | | | | | | | | | | | | +------->--------v------------+ +--------------<-+ +------v-------+ | impClickJoin | |impCountsByGender |impCountsByCountry | (parallelism=10) | |(parallelism=10) |(parallelism=10) ++-------------------+-+ +-----------+--+ +---+----------+ | | | | | | | | | | | | +--------v---------+ +-v-------------------+ | | |clickCountsByGender |clickCountsByCountry | | | |(parallelism=5) | |(parallelism=5) | | | +----+-------------+ +---------------------+ | | | | | | | | | | | | +----->+-----+>-----------v----+<---------------+ | report | |(parallelism=1) | +-----------------------+
(credit for above ascii art: http://www.asciiflow.com)
Done populating dummy data Executing filter task for filterImps_3 for impressions_demo Executing filter task for filterImps_2 for impressions_demo Executing filter task for filterImps_0 for impressions_demo Executing filter task for filterImps_1 for impressions_demo Executing filter task for filterImps_4 for impressions_demo Executing filter task for filterClicks_3 for clicks_demo Executing filter task for filterClicks_1 for clicks_demo Executing filter task for filterImps_8 for impressions_demo Executing filter task for filterImps_6 for impressions_demo Executing filter task for filterClicks_2 for clicks_demo Executing filter task for filterClicks_0 for clicks_demo Executing filter task for filterImps_7 for impressions_demo Executing filter task for filterImps_5 for impressions_demo Executing filter task for filterClicks_4 for clicks_demo Executing filter task for filterImps_9 for impressions_demo Running AggTask for impCountsByGender_3 for filtered_impressions_demo gender Running AggTask for impCountsByGender_2 for filtered_impressions_demo gender Running AggTask for impCountsByGender_0 for filtered_impressions_demo gender Running AggTask for impCountsByGender_9 for filtered_impressions_demo gender Running AggTask for impCountsByGender_1 for filtered_impressions_demo gender Running AggTask for impCountsByGender_4 for filtered_impressions_demo gender Running AggTask for impCountsByCountry_4 for filtered_impressions_demo country Running AggTask for impCountsByGender_5 for filtered_impressions_demo gender Executing JoinTask for impClickJoin_2 Running AggTask for impCountsByCountry_3 for filtered_impressions_demo country Running AggTask for impCountsByCountry_1 for filtered_impressions_demo country Running AggTask for impCountsByCountry_0 for filtered_impressions_demo country Running AggTask for impCountsByCountry_2 for filtered_impressions_demo country Running AggTask for impCountsByGender_6 for filtered_impressions_demo gender Executing JoinTask for impClickJoin_1 Executing JoinTask for impClickJoin_0 Executing JoinTask for impClickJoin_3 Running AggTask for impCountsByGender_8 for filtered_impressions_demo gender Executing JoinTask for impClickJoin_4 Running AggTask for impCountsByGender_7 for filtered_impressions_demo gender Running AggTask for impCountsByCountry_5 for filtered_impressions_demo country Running AggTask for impCountsByCountry_6 for filtered_impressions_demo country Executing JoinTask for impClickJoin_9 Running AggTask for impCountsByCountry_8 for filtered_impressions_demo country Running AggTask for impCountsByCountry_7 for filtered_impressions_demo country Executing JoinTask for impClickJoin_5 Executing JoinTask for impClickJoin_6 Running AggTask for impCountsByCountry_9 for filtered_impressions_demo country Executing JoinTask for impClickJoin_8 Executing JoinTask for impClickJoin_7 Running AggTask for clickCountsByCountry_1 for joined_clicks_demo country Running AggTask for clickCountsByCountry_0 for joined_clicks_demo country Running AggTask for clickCountsByCountry_2 for joined_clicks_demo country Running AggTask for clickCountsByCountry_3 for joined_clicks_demo country Running AggTask for clickCountsByGender_1 for joined_clicks_demo gender Running AggTask for clickCountsByCountry_4 for joined_clicks_demo country Running AggTask for clickCountsByGender_3 for joined_clicks_demo gender Running AggTask for clickCountsByGender_2 for joined_clicks_demo gender Running AggTask for clickCountsByGender_4 for joined_clicks_demo gender Running AggTask for clickCountsByGender_0 for joined_clicks_demo gender Running reports task Impression counts per country {CANADA=1940, US=1958, CHINA=2014, UNKNOWN=2022, UK=1946} Click counts per country {US=24, CANADA=14, CHINA=26, UNKNOWN=14, UK=22} Impression counts per gender {F=3325, UNKNOWN=3259, M=3296} Click counts per gender {F=33, UNKNOWN=32, M=35}