Google Git
Sign in
apache / arrow-experimental-rs-arrow2 / 12346c753d0d235121cb81e3d5f02859c8ad59b0
commit12346c753d0d235121cb81e3d5f02859c8ad59b0[log] [tgz]
authorFlorian Müller <florian@tomueller.de>Thu Apr 01 07:58:56 2021 -0400
committerAndrew Lamb <andrew@nerdnetworks.org>Thu Apr 01 07:58:56 2021 -0400
treefc15c08e69da76476081d543143003683a9173e3
parentb282d33f0cb81bed91d5b60592e169c79906f78e [diff]
ARROW-10354: [Rust][DataFusion] regexp_extract function to select regex groups from strings

Adds a regexp_extract compute kernel to select a substring based on a regular expression.

Some things I did that I may be doing wrong:

* I exposed `GenericStringBuilder`
* I build the resulting Array using a builder - this looks quite different from e.g. the substring kernel. Should I change it accordingly, e.g. because of performance considerations?
* In order to apply the new function in datafusion, I did not see a better solution than to handle the pattern string as `StringArray` and take the first record to compile the regex pattern from it and apply it to all values. Is there a way to define that an argument has to be a literal/scalar and cannot be filled by e.g. another column? I consider my current implementation quite error prone and would like to make this a bit more robust.

Closes #9428 from sweb/ARROW-10354/regexp_extract

Authored-by: Florian Müller <florian@tomueller.de>
Signed-off-by: Andrew Lamb <andrew@nerdnetworks.org>
4 files changed
tree: fc15c08e69da76476081d543143003683a9173e3
  1. .github/
  2. ci/
  3. dev/
  4. format/
  5. rust/
  6. testing
  7. .asf.yaml
  8. .clang-format
  9. .clang-tidy
  10. .clang-tidy-ignore
  11. .dir-locals.el
  12. .dockerignore
  13. .env
  14. .gitattributes
  15. .gitignore
  16. .gitmodules
  17. .hadolint.yaml
  18. .pre-commit-config.yaml
  19. .readthedocs.yml
  20. .travis.yml
  21. appveyor.yml
  22. CHANGELOG.md
  23. cmake-format.py
  24. CODE_OF_CONDUCT.md
  25. CONTRIBUTING.md
  26. docker-compose.yml
  27. header
  28. LICENSE.txt
  29. NOTICE.txt
  30. README.md
  31. run-cmake-format.py
README.md

Apache Arrow

Build Status Coverage Status Fuzzing Status License Twitter Follow

Powering In-Memory Analytics

Apache Arrow is a development platform for in-memory analytics. It contains a set of technologies that enable big data systems to process and move data fast.

Major components of the project include:

Arrow is an Apache Software Foundation project. Learn more at arrow.apache.org.

What's in the Arrow libraries?

The reference Arrow libraries contain many distinct software components:

  • Columnar vector and table-like containers (similar to data frames) supporting flat or nested types
  • Fast, language agnostic metadata messaging layer (using Google's Flatbuffers library)
  • Reference-counted off-heap buffer memory management, for zero-copy memory sharing and handling memory-mapped files
  • IO interfaces to local and remote filesystems
  • Self-describing binary wire formats (streaming and batch/file-like) for remote procedure calls (RPC) and interprocess communication (IPC)
  • Integration tests for verifying binary compatibility between the implementations (e.g. sending data from Java to C++)
  • Conversions to and from other in-memory data structures
  • Readers and writers for various widely-used file formats (such as Parquet, CSV)

Implementation status

The official Arrow libraries in this repository are in different stages of implementing the Arrow format and related features. See our current feature matrix on git master.

How to Contribute

Please read our latest project contribution guide.

Getting involved

Even if you do not plan to contribute to Apache Arrow itself or Arrow integrations in other projects, we'd be happy to have you involved: