Google Git
Sign in
apache / carbondata / f215eaad3b80f14032314f27c0a83f0ea77696a1
commitf215eaad3b80f14032314f27c0a83f0ea77696a1[log] [tgz]
authorakashrn5 <akashnilugal@gmail.com>Sat Mar 28 17:21:30 2020 +0530
committerajantha-bhat <ajanthabhat@gmail.com>Wed Apr 08 20:17:34 2020 +0530
tree5d42f2f8dbfd45eb506ab9a5b75a5a6a1466e2a4
parent8509af0b3a447715104b2ba3326d1762047615c3 [diff]
[CARBONDATA-3759] Refactor segmentRefreshInfo and fix cache issue in multiple application scenario

Why is this PR needed?
currently the segmentRefreshInfo is helping to clear the cache only in update cases and it fails to refresh the cache if any segment files changes or get updates.
when two applications are running on same store. One application changes some segment files changes and removes old cache and may be delete files, which wont be reflected in other application, which may result in either wrong results or query failure.

What changes were proposed in this PR?
refactor the segmentRefreshInfo to clear the cache and update when there are any updates on segments and if the segment files of respective segments changes.

Does this PR introduce any user interface change?
No

Is any new testcase added?
No
Tested in cluster and existing test cases will be enough.

This closes #3686
9 files changed
tree: 5d42f2f8dbfd45eb506ab9a5b75a5a6a1466e2a4
  1. .github/
  2. assembly/
  3. bin/
  4. build/
  5. common/
  6. conf/
  7. core/
  8. dev/
  9. docs/
  10. examples/
  11. format/
  12. geo/
  13. hadoop/
  14. index/
  15. integration/
  16. licenses-binary/
  17. mv/
  18. processing/
  19. python/
  20. sdk/
  21. streaming/
  22. tools/
  23. .gitignore
  24. LICENSE
  25. NOTICE
  26. pom.xml
  27. README.md
README.md

Apache CarbonData is an indexed columnar data store solution for fast analytics on big data platform, e.g.Apache Hadoop, Apache Spark, etc.

You can find the latest CarbonData document and learn more at: http://carbondata.apache.org

CarbonData cwiki

Visit count: HitCount

Status

Spark2.3: Build Status Coverage Status

Features

CarbonData file format is a columnar store in HDFS, it has many features that a modern columnar format has, such as splittable, compression schema ,complex data type etc, and CarbonData has following unique features:

  • Stores data along with index: it can significantly accelerate query performance and reduces the I/O scans and CPU resources, where there are filters in the query. CarbonData index consists of multiple level of indices, a processing framework can leverage this index to reduce the task it needs to schedule and process, and it can also do skip scan in more finer grain unit (called blocklet) in task side scanning instead of scanning the whole file.
  • Operable encoded data :Through supporting efficient compression and global encoding schemes, can query on compressed/encoded data, the data can be converted just before returning the results to the users, which is “late materialized”.
  • Supports for various use cases with one single Data format : like interactive OLAP-style query, Sequential Access (big scan), Random Access (narrow scan).

Building CarbonData

CarbonData is built using Apache Maven, to build CarbonData

Online Documentation

Integration

Other Technical Material

Fork and Contribute

This is an active open source project for everyone, and we are always open to people who want to use this system or contribute to it. This guide document introduce how to contribute to CarbonData.

Contact us

To get involved in CarbonData:

About

Apache CarbonData is an open source project of The Apache Software Foundation (ASF).