tree 0c528cd4c8a6a4cedf48c8485de6d110a9d50479
parent 350663f61842ede34f06224aec92e38d1bff92ee
author chitralverma <chitralverma@gmail.com> 1574299604 +0800
committer William Guo <guoyp@apache.org> 1574299604 +0800

[GRIFFIN-297] Allow support for additional file based data sources

**What changes were proposed in this pull request?**

The PR extends the current support beyond just Avro and Text for various file based data sources (Parquet, ORC, etc).

 - Allows users to specify additional file based data sources like Parquet, CSV, TSV, ORC etc.
 - Allows data to be read directly from stand-alone files as well as directories present in both local/ distributed file systems.
 - Allows users to specify schema directly through options (useful for CSV/ TSV types).

A sample config looks like,

```
{
  "name": "source",
  "baseline": true,
  "connectors": [
    {
      "type": "file",
      "version": "1.7",
      "config": {
        "format": "parquet",
        "options": {
          "k1": "v1",
          "k2": "v2"
        },
        "paths": [
          "/home/chitral/path/to/source/",
          "/home/chitral/path/to/test.parquet"
        ]
      }
    }
  ]
}

```
**Does this PR introduce any user-facing change?**
No

**How was this patch tested?**
Griffin test suite. Some additional unit test has also been added.

Author: chitralverma <chitralverma@gmail.com>

Closes #555 from chitralverma/allow_file_based_batch_connectors.