| commit | 62d755d6f0474d4492d598d469e02c891d99bf95 | [log] [tgz] |
|---|---|---|
| author | wankunde <wankunde@163.com> | Mon Jul 08 21:19:55 2019 +0800 |
| committer | William Guo <guoyp@apache.org> | Mon Jul 08 21:19:55 2019 +0800 |
| tree | 9eebd9b905249f53da057053681de4c75314ad51 | |
| parent | 42bd3e111db4b41e5ac23a14021d22973801c7db [diff] |
build and run transform steps with multiple threads There are sequence of dq steps in each DQJob,and run those steps one by one (with foldLeft function). We can use multiple threads to run some of those steps which have no dependency. For example: In a DQBatchJob, a accuracyExpr will have for steps **_missRecords ,_missCount , __totalCount, accu** . _missCount and **_totalCount step can run at the same time . In SeqDQStep ,it just need contains some root steps without dependency steps. If each step knows it's dependency steps, and when they are ready, we can run the step itself . Running step : accu | |---__missCount | | |---__missRecords | |---__totalCount Running step : __missCount | |---__missRecords Running step : __missRecords Running step : __totalCount Author: wankunde <wankunde@163.com> Closes #504 from wankunde/master.
The data quality (DQ) is a key criteria for many data consumers like IoT, machine learning etc., however, there is no standard agreement on how to determine “good” data. Apache Griffin is a model-driven data quality service platform where you can examine your data on-demand. It provides a standard process to define data quality measures, executions and reports, allowing those examinations across multiple data systems. When you don't trust your data, or concern that poorly controlled data can negatively impact critical decision, you can utilize Apache Griffin to ensure data quality.
You can try running Griffin in docker following the docker guide.
Follow Apache Griffin Development Environment Build Guide to set up development environment.
If you want to contribute codes to Griffin, please follow Apache Griffin Development Code Style Config Guide to keep consistent code style.
If you want to deploy Griffin in your local environment, please follow Apache Griffin Deployment Guide.
For more information about Griffin, please visit our website at: griffin home page.
You can contact us via email:
You can also subscribe the latest information by sending a email to subscribe dev-list and subscribe user-list. You can also subscribe the latest information by sending a email to subscribe dev-list and user-list:
dev-subscribe@griffin.apache.org users-subscribe@griffin.apache.org
You can access our issues on JIRA page
See How to Contribute for details on how to contribute code, documentation, etc.
Here's the most direct way to contribute your work merged into Apache Griffin.