tree d94c1636a9a44f52cf40ef137d6a5ec9d2c29462
parent cd28146e4b4ad59587f2ad4c2748905954f837d8
author Maria Podorvanova <36038640+podorvanova@users.noreply.github.com> 1628706142 +1000
committer GitHub <noreply@github.com> 1628706142 +0530
gpgsig -----BEGIN PGP SIGNATURE-----
 
 wsBcBAABCAAQBQJhFBVeCRBK7hj4Ov3rIwAAztQIAA15zNUb6nDBiiUO9hQGvi5r
 KKU+KILSV0wt+xUg+LqLKCOhgvWR/AXCMqTeHTPAFsfimtSWD7zbzBDrXBwxAcxv
 KHzrB4yogq3693KPsBjJw7WLrox8Jbrpnui5OR+8x8KlTyrRTueG178/Fn8koY8Z
 Y4sM3ab33jxYXHadn9XYrYX53KeoQ3kvYjGzXaOYn3Ydi5vtkl1BnxRYYdCh4KlH
 VwGAI5wI6euGm3zLLvwqWTFtJ/is99jJUS0HlXAw5jt6ZrafprJf5+SxQvfRJHEO
 vnA0eyKYiw2sQzmqDr9KwPM7VRAoxsOPdO9zPEVFiGDkc9Ccp3jmG3y7NsMP6b4=
 =zXjw
 -----END PGP SIGNATURE-----
 

GORA-664 Add datastore for Elasticsearch (#234)

* Create basic gora-elasticsearch module

* Bump Elasticsearch version and remove redundant dependency

* Implement connection and basic schema management

- Create ElasticsearchStore class with connection initialization
- Create basic Elasticsearch types mapping
- Implement the necessary files for mapping representation (ElasticsearchMapping, ElasticsearchMappingBuilder)
- Read schema from mapping file
- Cover initialization with test

* Set up Elasticsearch client parameters

- Created gora.properties file with configuration properties
- Loaded connection parameters from configuration
- Implemented connection to Elasticsearch cluster with ElasticsearchParameters
- Covered ElasticsearchParameters with tests
- Added javadoc descriptions

* Add a property for choosing the authentication method

* Implement testing with Elasticsearch container

- Added testing dependencies
- Added GoraElasticsearchTestDriver with Elasticsearch container
- Added javadoc descriptions to GoraElasticsearchTestDriver class
- Fixed two existing tests in accordance to Elasticsearch container

* Implement some methods for schema management

Implemented schemaExists, createSchema, deleteSchema and flush methods

* Add XSD validation file for the XML mapping

* Fix XSD validation

- Relocated gora-elasticsearch.xsd file to main resources
- Covered XSD validation with test
- Added gora-elasticsearch-mapping-invalid.xml file for test

* Set up Elasticsearch container's authentication parameters

* Implement exists method

* Add comments for the connection parameters

* Fix authentication

- Set up password to Elasticsearch container properly
- Set default Elasticsearch container server’s username in gora.properties
- Added exceptions for missing arguments in authentication

* Add parameter for the XSD validation

- Defined a parameter for the XSD validation
- Added a test case for the parameter
- Made ElasticsearchStore read mapping file from properties, not configuration

* Implement some basic Input-Output operations for schema management

- Implemented delete, get and put methods
- Implemented newInstance and getUnionSchema utility methods
- Implemented basic serialization/deserialization for primitive AVRO types

* Fix createSchema method

- Added mappings while creating an Elasticsearch index
- Added getter and setter to Datatype enum

* Implement serialization/deserialization for some Avro data types

- Implemented serializeFieldValue and deserializeFieldValue methods for ARRAY, BOOLEAN, BYTES and FIXED Avro data types
- Fixed deserialization for STRING Avro data type
- Added javadoc descriptions

* Fix NPE when getting a non-existent Elasticsearch document

* Implement serialization/deserialization for MAP Avro data type

* Refactor serialization/deserialization to have better javadocs and arguments

* Implement serialization/deserialization for RECORD Avro data type

* Implement serialization/deserialization for UNION Avro data type

* Fix passed Schema argument for ARRAY deserialization

* Fix BYTES deserialization for Base64 encoded String

* Ignore testGet3UnionField test

* Add javadoc descriptions to serialization and deserialization methods

* Implement newQuery method

* Implement deleteByQuery method

* Use an Enum instead of literal strings for the Authentication Type parameter

* Use parameterized logging instead of string concatenation

* Implement execute method

* Implement getPartitions method

* Add scaling_factor support

* Remove unsupported Elasticsearch data types

* Implement Metadata Analyzer for Elasticsearch Store

* Try to fix range query by “_id” field

* Fix execute method by adding a special "gora_id" field

* Implement deleting specific fields of the records in deleteByQuery method

* Implement MapReduce test

* Fix flush method by using refresh

* Address reviewer's comments

* Add Elasticsearch specific logging dependency