tree a326a2f9b0c0cd560fcdcf8503849af9917dc8fc
parent 4c90c0c21d8271b7908122e6104e85a12cd6c248
author Agustin Gonzalez <agustin.gonzalez@imply.io> 1625814629 -0700
committer GitHub <noreply@github.com> 1625814629 -0700
gpgsig -----BEGIN PGP SIGNATURE-----
 
 wsBcBAABCAAQBQJg5/ZlCRBK7hj4Ov3rIwAAkc0IAFf77/LXdaAT8g54LgqKLnht
 J+emqkFcnDGeOChle7+9syr8uq+YrgA59wBD5PtzrKP810XwqkVnYPe8KwU/I7HS
 kfixP9BO05+5lAcHgVdyWXJRLser0PqSfThHl2bGBslpk4xN/VKYXfLCoQkz0w5C
 IaWjZZX4p6AsDnW5gbncpPJklUml0WFmacGUXyqE3LFx3oPLsfLITdBt34hTAYrV
 E49UfB0XMU4RQc/Dfrp7tEoA2PSbR+1w8YJq0lPLLdRyK/xc3NjpNWoKgH1kRKcQ
 //TB/ZjDzrJlFb/QDq4yuo97gWL7rAqhEb8a6eFr349CVnQbHqqsccSzWd6ZA/E=
 =HcT2
 -----END PGP SIGNATURE-----
 

Bound memory utilization for dynamic partitioning (i.e. memory growth is constant) (#11294)

* Bound memory in native batch ingest create segments

* Move BatchAppenderatorDriverTest to indexing service... note that we had to put the sink back in sinks in mergeandpush since the persistent data needs to be dropped and the sink is required for that

* Remove sinks from memory and clean up intermediate persists dirs manually after sink has been merged

* Changed name from RealtimeAppenderator to StreamAppenderator

* Style

* Incorporating tests from StreamAppenderatorTest

* Keep totalRows and cleanup code

* Added missing dep

* Fix unit test

* Checkstyle

* allowIncrementalPersists should always be true for batch

* Added sinks metadata

* clear sinks metadata when closing appenderator

* Style + minor edits to log msgs

* Update sinks metadata & totalRows when dropping a sink (segment)

* Remove max

* Intelli-j check

* Keep a count of hydrants persisted by sink for sanity check before merge

* Move out sanity

* Add previous hydrant count to sink metadata

* Remove redundant field from SinkMetadata

* Remove unneeded functions

* Cleanup unused code

* Removed unused code

* Remove unused field

* Exclude it from jacoco because it is very hard to get branch coverage

* Remove segment announcement and some other minor cleanup

* Add fallback flag

* Minor code cleanup

* Checkstyle

* Code review changes

* Update batchMemoryMappedIndex name

* Code review comments

* Exclude class from coverage, will include again when packaging gets fixed

* Moved test classes to server module

* More BatchAppenderator cleanup

* Fix bug in wrong counting of totalHydrants plus minor cleanup in add

* Removed left over comments

* Have BatchAppenderator follow the Appenderator contract for push & getSegments

* Fix LGTM violations

* Review comments

* Add stats after push is done

* Code review comments (cleanup, remove rest of synchronization constructs in batch appenderator, reneame feature flag,
remove real time flag stuff from stream appenderator, etc.)

* Update javadocs

* Add thread safety notice to BatchAppenderator

* Further cleanup config

* More config cleanup