Fix AO row scan expanding unread columns (#965) Closes field-engineering/dev_collaboration#965

## Summary

Fixes field-engineering/dev_collaboration#965: a `count(1)` over a 281 column
append-only **row** table, qualified on three date columns, was 6x slower than the same
data in a heap table and 14x slower than AOCS.

### Root cause

`AppendOnlyExecutorReadBlock_ProcessTuple()` called `memtuple_deform()` and then
`ExecStoreVirtualTuple()`, and `appendonly_slot_callbacks()` returned `TTSOpsVirtual`.
A virtual slot promises its values are already in `tts_values[]`, so
`ExecStoreVirtualTuple()` sets `tts_nvalid = natts` and `slot_getattr()` never reaches
`getsomeattrs` — the scan has no choice but to expand **every** column of **every**
tuple. Heap tables are unaffected because `TTSOpsBufferHeapTuple` deforms incrementally
up to the highest attribute number the query references.

Scan cost therefore grew with the **width of the table** instead of the columns actually
read. Measured on a release build (`-O3`, no cassert), 500k rows, query touching only
attno 2:

| table columns | AO row before | heap |
|---|---|---|
| 20 | 46.9 ms | 35.7 ms |
| 100 | 114.2 ms | 36.8 ms |
| 281 | **236.0 ms** | 39.1 ms |

This is not how the format is meant to be read. A MemTuple keeps precomputed attribute
offsets in its binding, so any attribute is reachable in constant time, and before the
PostgreSQL 12 merge (`19cd1cf4b6`) the slot carried that binding and fetched attributes
on demand. That merge dropped the MemTuple slot shape — which at the time could only
exist as extra fields in the shared `TupleTableSlot` struct — and converted the tuple
eagerly instead. PostgreSQL 12 also introduced `TupleTableSlotOps` precisely so an access
method can keep tuples in its native format, which is what this MR finally uses.

**The same code is present in GPDB 7 and Apache Cloudberry main**, both verified to show
the identical regression (GPDB 7 release: 193.8 ms vs heap 28.9 ms at 281 columns). This
is an inherited PG12-merge artifact, not something introduced by Lightning.

### Change

Add `TTSOpsMemTuple`, a slot holding the MemTuple together with the binding needed to
interpret it, expanding attributes in `getsomeattrs`. Because offsets are precomputed,
`tts_nvalid` alone is the resume point — no separate deforming cursor (the `off` field
heap and minimal tuple slots carry) is needed. `memtuple_deform_range()` reuses the
existing inlinable internal helper so full-column access keeps the old bulk-deform speed.

Two details are easy to miss when adding a slot type here, both found by running the full
regression suite:

- `gp_segment_id` and `gp_foreign_server` are answered by the slot even though no tuple
  format carries them, so `getsysattr` must keep the special cases that
  `tts_virtual_getsysattr` documents. Omitting them broke ~59 tests.
- Not every slot reaches the scan through `table_slot_create()`.
  `AppendOnlySegmentFileFullCompaction()` built a virtual slot directly and passed it to
  `appendonly_getnextslot()`; in a release build the assertion is compiled out and the
  larger slot struct would be written past the end of the smaller one. It now asks the
  access method for the slot type, and `ProcessTuple` additionally falls back to expanding
  the tuple up front for any non-MemTuple slot, so a caller bringing its own slot cannot
  be handed one it cannot read.

## Test plan

Release build (`-O3`, `debug_assertions=off`), demo cluster, 500k rows.

The two tables below pin one variable each. Query is
`SELECT count(1) FROM t WHERE c2 > 0`, i.e. it references attno 2 only, unless stated.

**1. Cost no longer depends on how wide the table is.** Query fixed on attno 2, table width varied:

| table columns | before | after | heap |
|---|---|---|---|
| 20 | 46.9 ms | 35.9 ms | 35.7 ms |
| 50 | 67.3 ms | 36.9 ms | 38.4 ms |
| 100 | 114.2 ms | 36.0 ms | 36.8 ms |
| 200 | 171.1 ms | 37.6 ms | 39.0 ms |
| **281** | **236.0 ms** | **41.4 ms** | 39.1 ms |
| 281, all columns populated | 415.0 ms | 112.7 ms | 103.6 ms |

**2. Cost now depends on the highest attribute number referenced, exactly as heap does.**
Table fixed at 281 columns, referenced column varied (`count(cN)`):

| referenced up to | after | heap |
|---|---|---|
| c2 | 37.3 ms | 36.1 ms |
| c50 | 68.2 ms | 56.4 ms |
| c140 | 121.5 ms | 95.4 ms |
| c281 | 212.4 ms | 156.2 ms |

Slopes: AO 0.63 ms per column per 500k rows, heap 0.43 ms. So the win is not that MemTuple
deforms faster than heap — it does not — it is that unread columns are no longer deformed at
all. A query reaching the last column costs what it always cost; the flat line in table 1 is
a consequence of referencing attno 2, not an unconditional property.

The reported query references attno 9, 12 and 13, so it now expands 13 columns instead of
281. `getsomeattrs` fills a prefix, so the gap columns (1-8, 10, 11) are still expanded;
skipping those would need per-column projection, which `tts_nvalid` cannot express (it is a
single watermark, so validity is necessarily a prefix) and which would mean claiming
unfilled columns are valid. Not attempted here.

Full-column access is not penalised: `count(c281)` 217.7 ms (before the change a bare scan
already cost 236.0 ms because it deformed everything), `sum(c281)` on fully populated data
415.9 ms vs 415.0 ms.

**Correctness**
- [x] `SELECT * FROM ao EXCEPT SELECT * FROM heap` and the reverse both return 0 rows, over
      all 281 columns, for sparse and fully populated data
- [x] System columns unchanged: `ctid` / `tableoid` / `gp_segment_id` work, the four MVCC
      columns still error (a MemTuple carries no MVCC fields; visibility lives in the visimap)
- [x] Materialize-dependent paths: CTE MATERIALIZED, ORDER BY, CTAS round trip, cursor
      across statements
- [x] `make installcheck` A/B on the same cluster: **baseline 36/723 failed, with this
      change 36/723 failed, failure sets identical**. The 36 are environmental in this
      build (no pax_storage / iceberg AM, ORCA disabled).
- [x] Builds clean under `-O3 -Werror`

**Not done**
- [ ] No regression test added for the wide-table-narrow-projection behaviour. Both bugs
      above surfaced only from the full suite, so this path is under-covered; a test
      exercising a wide table plus the VACUUM / CLUSTER / CREATE INDEX paths would be
      worth adding. Happy to add it to this MR if preferred.
- [ ] Not yet verified on GPDB 7 / upstream Cloudberry, though the patch applies to the
      identical code there.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

(cherry picked from commit 47a358712540acfa3ecd7ee1cb778ed01d5298f4)
8 files changed
tree: 1403d58c287be74838363c2ee17ec6cd85c4bd28
  1. .abi-check/
  2. .github/
  3. config/
  4. contrib/
  5. devops/
  6. doc/
  7. gpAux/
  8. gpcontrib/
  9. gpMgmt/
  10. licenses/
  11. mcp-server/
  12. src/
  13. .asf.yaml
  14. .clang-tidy
  15. .dir-locals.el
  16. .editorconfig
  17. .git-blame-ignore-revs
  18. .gitattributes
  19. .gitignore
  20. .gitmessage
  21. .gitmodules
  22. aclocal.m4
  23. AGENTS.md.template
  24. AI_GUIDELINE.md
  25. CODE_OF_CONDUCT.md
  26. configure
  27. configure.ac
  28. CONTRIBUTING.md
  29. DISCLAIMER
  30. getversion
  31. GNUmakefile.in
  32. LICENSE
  33. Makefile
  34. meson.build
  35. meson_options.txt
  36. NOTICE
  37. pom.xml
  38. putversion
  39. python-dependencies.txt
  40. README.apache.md
  41. README.md
  42. SECURITY.md
  43. sonar-project.properties
README.md

Apache Cloudberry (Incubating)

Website Documentation Slack Twitter Follow WeChat Youtube GitHub Discussions GitHub commit activity(branch) GitHub contributors GitHub License Apache Cloudberry Build Ask DeepWiki Apache Rat Audit

Introduction

Apache Cloudberry (Incubating), created by the original developers of Greenplum® Database, is one advanced and mature open-source Massively Parallel Processing (MPP) database, which evolves from the open-source Greenplum® Database but features a newer PostgreSQL kernel and more advanced enterprise capabilities. It can serve as a data warehouse and can also be used for large-scale analytics and AI/ML workloads.

Build and try out

Build from source

You can follow these guides to build Cloudberry on Linux OS (including RHEL/Rocky Linux, and Ubuntu) and macOS.

Try out quickly

Welcome to try out Cloudberry via building one Docker-based Sandbox, which is tailored to help you gain a basic understanding of Cloudberry's capabilities and features.

Repositories

This is the main repository for Apache Cloudberry (Incubating). Alongside this, there are several ecosystem repositories for Cloudberry, including the website, extensions, connectors, adapters, and other utilities.

Community & Support

We have many channels for community members to discuss, ask for help, feedback, and chat:

TypeDescription
SlackClick to Join the real-time chat on Slack for QA, Dev, Events, and more. Don't miss out! Check out the Slack guide to learn more.
Q&AAsk for help when running/developing Cloudberry, visit GitHub Discussions - QA.
New ideas / Feature RequestsShare ideas for new features, visit GitHub Discussions - Ideas.
Report bugsProblems and issues in Apache Cloudberry core. If you find bugs, welcome to submit them here.
Report a security vulnerabilityView our security policy to learn how to report and contact us.
Community eventsIncluding meetups, webinars, conferences, and more events, visit the Events page and subscribe to the events calendar.
DocumentationOfficial documentation for Cloudberry. You can explore it to discover more details about us.

Contribution

Contributions can be diverse, such as code enhancements, bug fixes, feature proposals, documents, marketing, and so on. No contribution is too small, we encourage all types of contributions. Cloudberry community welcomes contributions from anyone, new and experienced! Our contribution guide will help you get started with the contribution.

TypeDescription
Code contributionLearn how to contribute code to the Cloudberry, including coding preparation, conventions, workflow, review, and checklist following the code contribution guide.
Submit the proposalProposing major changes to Cloudberry through proposal guide.
Doc contributionWe need you to join us to help us improve the documentation, see the doc contribution guide.
AI guidlineFor AI-assisted development, please review our AI guideline for advice on responsible AI usage.

Roadmap

You can check our Cloudberry Roadmap out to see the product plans and goals we want to achieve. Welcome to share your thoughts and ideas to join us in shaping the future of Apache Cloudberry (Incubating). (We will update the Roadmap after entering the Incubator.)

Acknowledgment

Thanks to PostgreSQL, Greenplum Database and other great open source projects to make Apache Cloudberry has a sound foundation.

License

Cloudberry is licensed under the Apache License, Version 2.0. For details, see the LICENSE.

Crypto Export Notice

This distribution includes cryptographic software. The country in which you currently reside may have restrictions on the import, possession, use, and/or re-export to another country, of encryption software. BEFORE using any encryption software, please check your country's laws, regulations and policies concerning the import, possession, or use, and re-export of encryption software, to see if this is permitted. See http://www.wassenaar.org for more information.

The Apache Software Foundation has classified this software as Export Commodity Control Number (ECCN) 5D002, which includes information security software using or performing cryptographic functions with asymmetric algorithms. The form and manner of this Apache Software Foundation distribution makes it eligible for export under the “publicly available” Section 742.15(b) exemption (see the BIS Export Administration Regulations, Section 742.15(b)) for both object code and source code.

The following provides more details on the included cryptographic software:

  • Apache Cloudberry uses OpenSSL for TLS support on client and server connections, and for the SCRAM-SHA-256 authentication method.
  • Apache Cloudberry optionally uses GSSAPI (Kerberos) for authentication and connection encryption, and LDAP over TLS for authentication.
  • The bundled pgcrypto extension provides cryptographic functions to SQL, using either OpenSSL or its own built-in implementations.
  • The gpfdist external table file server can serve over HTTPS, using OpenSSL and X.509 certificates supplied through its --ssl <certificates_directory> option.

Apache Cloudberry does not ship OpenSSL itself; it links against the OpenSSL provided by the operating system.

ASF Incubator disclaimer

Apache Cloudberry is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required for all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.