feat: migrate Paimon C++ 0.3 development history (#161)

* feat: Add metadata system tables

* fix: add break for CHAR_END_SUBTYPE

* fix: Fix MemorySegment::Compare to use big-endian byte-order comparison semantics

* chore(license): refine build support notices

* fix: Fix undefined behavior in BinarySection bitmask constants caused by signed left shift

* refactor: ColumnarArray: change constructor parameter to raw pointer to clarify non-owning semantics

* feat: Support macOS builds with AppleClang and Homebrew Clang

* fix: fix undefined behavior in bitwise left-shift operations across codebase

* feat: optimize parquet reads with page-level filtering

* fix: add ScopeGuard to wait async tasks before early return in OrphanFilesCleanerImpl::Clean()

* feat(lumina): support null values in vector column during index building

* chore: modify comment in RecordBatch & add check in GenericRow

* fix: fix clang-tidy issues

* chore: add check scale in BinaryRowWriter

* fix: fix ub in Blob

* refact: Refactor Literal operator== to delegate to CompareTo for consistent equality semantics

* refactor: Reuse RowGroupPageIndexReader across columns to improve page-level predicate pushdown performance

* feat(blob): Support blob-descriptor-field for inline blob descriptor storage

* fix: Like::TestString to align with Java LIKE semantics

* chore: update LICENSE with additional PyTorch copyright information

* fix: date validation reject invalid dates instead of silent normalization

* fix: fix thread-safe problem for file store path factory

* feat: Improve PK MOR read performance with batch queue and cached column metadata

* fix: avoid signed integer overflow UB in BloomFilter and DeltaVarintCompressor

* feat: optimize count(*) for pk/append table

Co-authored-by: dalingmeng <menglingda.mld@alibaba-inc.com>

* feat(blob): support blob-view-field for cross-table blob reference re…

* fix: fix unstable prefetch reader case

* feat: remove support for FLOAT/DOUBLE type partition field

* fix: skip object store check when UseRESTCatalogCommit is enabled

* feat: add Bucket() virtual interface to DataSplit base class

* fix(fs): normalize local filesystem paths and some refactoring

* fix(fs): normalize local paths

* fix(fs): simplify external path position update

* fix(schema): treat rowkind as special field

* refactor(fs): return LocalFile create as unique ptr

* feat(benchmark): add append/pk table benchmark

* Allow FileStoreCommit for PK tables with postpone bucket mode

* Allow FileStoreCommit for PK tables with postpone bucket mode

Postpone bucket mode (bucket=-2) writes data like an append table:
all files go to bucket--2/ directory and the REST catalog server
handles bucket redistribution during background compaction. The
commit logic (manifest and snapshot generation) is identical to
append tables, so there is no reason to block it.

See: https://paimon.apache.org/docs/master/primary-key-table/data-distribution/#postpone-bucket

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* Address review feedback: use NumBuckets() and Options::BUCKET constant

- Replace raw schema options map lookup with TableSchema::NumBuckets()
  for postpone bucket check
- Use Options::BUCKET constant instead of hardcoded "bucket" in tests
- Add IsNotImplemented() status kind assertion in rejection test

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* fix: Avoid signed overflow UB in BSI index and null-deref in bloom filter index

* fix: LZ4 block compressor/decompressor safety issues

* fix(avro): format avro types in decoder errors

* fix: override CachedInputStream::Advance to avoid real I/O for skipped pages

* fix: replace arrow type DECIMAL to DECIMAL128

* fix: remove trailing semicolons from statement-like macro definitions

* fix: change FileStorePathFactory::Create() to return shared_ptr

* refactor(blob): refactor BlobFileContext to honor the actual schema when classifying BLOB fields

* fix: address ubsan findings

* fix(build): make package config relocatable

* fix(parquet): fix incorrect chunk_end calculation in ComputePageRanges when dictionary page is present

* feat(fs): support int64 file IO sizes

* fix: small fixes for parquet reader including RowRanges, PreBuffer, ColumnIndexFilter and code cleanup

* feat(blob view): support multi-thread reading upstream table & add inte test

* test: add blob compatible test

* refractor(parquet): simplify FileReaderWrapper and PageFilteredRowGroupReader with unified TargetRowGroup structure

* fix(scan): fix dangling reference and unsigned overflow in FileStoreScan

* feat(shredding): Add infrastructure for shared-shredding MAP storage layout

* fix: make byte hashing independent of char signedness

* feat(shared-shredding): support nested field projection in ORC format reader

* fix: prevent zero-thread DefaultExecutor and validate FileSystem/SchemeMap conflict

* fix: small fixes for AI reviewing feedback

* feat: Optimize PK MOR read performance by reduce per-row object allocation and improve merge read performance.

* feat: Add manifests and files system tables

* fix: canonicalize NaN for Hive bucket hash

* feat(cache): support manifest file cache

* feat(schema): support pk table schema evolution

* fix: fix min/max row id serialize bug

* feat: update lumina lib to v0.3.0-rc1

* feat(shredding): support shared-shredding map write

* fix(parquet): fallback nested field reading to RowGroup reading (walkaround)

* feat(tantivy): add Tantivy full-text global index via Rust FFI

* feat(parquet): support parquet metadata cache

* fix(format): reset avro reader on schema change

* feat(shredding): support shared shredding file reader

* fix: address avro blob and executor review issues

* feat(shredding): improve shared-shredding adaptive width restore & refactor write

* feat: support read for nested type sub column

* fix: close input/output streams when remote lookup file read/write failed

* fix(testing): address review issues

* feat(shared_shredding): add read inte test for shared_shredding

* feat: add CreateReader api with field_name & index_type

* feat(shredding): add shared-shredding map placement policies

* feat(types): support CHAR/VARCHAR/BINARY/VARBINARY in data type json parser

* fix: align duplicate-key replacement semantics with Java Paimon

* fix: handle non-contiguous RowRanges when resolving global row IDs

* test: add tests for executor

* fix: release global index writer before reader to avoid mem issue when write index failed

* fix: fix IsThreadSafe() in UnionGlobalIndexReader

* test: add map shared-shredding tests for compaction and alter table and predicates

* feat(blob): Remove blob external storage mode

* feat(schema): scan context support set table schema

* feat(manifest): support snapshot live manifest cache

* feat(parquet): support pushing bitmaps down to page-level filtering (except for nested columns)

* chore: adjust set read_ranges_freshed_ in SetReadRanges

* docs: add coding agent guidelines

* fix: compile with correct flags with AppleClang

* feat(shared-shredding): Expose shared-shredding utilities

* feat(blob view): support blob-view.resolve.enabled to preserve blob view references at read time

* fix(build): disable glog unwind without libunwind

* fix(avro): include headers used by file batch reader

* fix(orc): avoid removed Arrow bitmap append API

* feat(commit): align functionality with java paimon commit

* feat: add read-optimized system table

* feat(blob): support blob.split-by-file-size to weigh blob files in scan splitting

* feat(blob): support blob-write-null-on-missing-file and blob-write-null-on-fetch-failure options

* fix(commit): fix changelogRecordCount nullopt and add ut

* fix(fs): create parent directories before opening Jindo writers

* fix(jindo): make async reads concurrency-safe

* fix: enable TSAN and resolve detected data races

* feat(parquet): support configurable bitmap row-range refining strategies (coalesce and trim)

* feat(commit): support TruncateTable/Abort/RollbackToAsLatest and add ut

* build: fetch Lumina from release package

* fix(blob): identify a missing blob file with FileSystem::Exists

* fix(btree): preserve empty keys in index metadata & disable global bitmap index

* feat(variant): support variant data type

* build: fetch Boost from OSS and define network test macro globally

* build: fetch Boost from OSS

* build: define network test macro globally

* feat: add metrics storage-read-bytes for parquet and fix clean inte test

* feat(lumina): support Lumina tag filtering

* fix(reader): ensure batch readers return zero-offset Arrow arrays

* feat: add global system tables framework under `sys` database

* refactor(format): remove Lance support

* refactor(tantivy): move FFI crate under crates

* chore: clean up attribution and test data whitespace

* fix(blob): support blob-only writes and reject nested blob fields

* fix: nested list schema evolution

* fix(abi): dynamic_cast error on Predicate across shared libraries

* feat(fs): add object store and S3 file systems

Add a reusable read-only object store layer with a curl-based HTTP
transport.

Use the AWS C authentication components for credential resolution and
request signing while keeping S3 data access independent of the full AWS
SDK.

* feat(parquet): support page-level (page index) filtering for nested column types (patch arrow)

* test: add test for test coverage

* test: add ut for release-0.3

* chore: remove Git LFS and refresh documentation

* fix(blob): allow blob files across schema IDs

* fix(core): support fallback keys for ignore-delete option

* feat(shredding): support adaptive schemas across rolling files

* fix(lucene): normalize prefix and wildcard queries

* chore: update version to 0.3.0

* feat(blob): support placeholder fallback for partial updates

* feat(parquet): support configuring Arrow pre-buffer hole-size-limit for range coalescing

* fix: align floating-point semantics with Java

* chore: remove target-only migration leftover

* fix(release): align documentation version

* ci: add consolidated build and test workflow

* fix: align source archive and integration tests

* ci: disable clang-tidy

---------

Co-authored-by: Socrates <suxiaogang223@icloud.com>
Co-authored-by: lszskye <57179283+lszskye@users.noreply.github.com>
Co-authored-by: Zhang Jiawei <30893610+zjw1111@users.noreply.github.com>
Co-authored-by: liangjie <liangjie.liang@antgroup.com>
Co-authored-by: Zhou Hongfeng <87103887+zhf999@users.noreply.github.com>
Co-authored-by: dalingmeng <49717204+dalingmeng@users.noreply.github.com>
Co-authored-by: Yonghao Fang <yonghao.fyh@alibaba-inc.com>
Co-authored-by: dalingmeng <menglingda.mld@alibaba-inc.com>
Co-authored-by: Joey <liujiayi771@apache.org>
Co-authored-by: fourier307 <8501328+fourier307@users.noreply.github.com>
Co-authored-by: Joey <joey.ljy@alibaba-inc.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Gang Wu <ustcwg@gmail.com>
Co-authored-by: gripleaf <425797155@qq.com>
Co-authored-by: spaces-x <dlut.weixiang@gmail.com>
Co-authored-by: Nicholas Jiang <programgeek@163.com>
Co-authored-by: Xiaoguang Zhu <smdsbz@qq.com>
Co-authored-by: Mr Dk. <mrdrivingduck@gmail.com>
Co-authored-by: Zouxxyy <zouxinyu.zxy@alibaba-inc.com>
Co-authored-by: XiaoHongbo <1346652787@qq.com>
Co-authored-by: Wei Zhang <zhangweilst@gmail.com>