| <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "https://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
| <html xmlns="http://www.w3.org/1999/xhtml" lang="en-US"> |
| <head> |
| <meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/> |
| <meta http-equiv="X-UA-Compatible" content="IE=11"/> |
| <meta name="generator" content="Doxygen 1.9.8"/> |
| <meta name="viewport" content="width=device-width, initial-scale=1"/> |
| <title>iceberg-cpp: README</title> |
| <link href="tabs.css" rel="stylesheet" type="text/css"/> |
| <script type="text/javascript" src="jquery.js"></script> |
| <script type="text/javascript" src="dynsections.js"></script> |
| <link href="search/search.css" rel="stylesheet" type="text/css"/> |
| <script type="text/javascript" src="search/searchdata.js"></script> |
| <script type="text/javascript" src="search/search.js"></script> |
| <link href="doxygen.css" rel="stylesheet" type="text/css" /> |
| </head> |
| <body> |
| <div id="top"><!-- do not remove this div, it is closed by doxygen! --> |
| <div id="titlearea"> |
| <table cellspacing="0" cellpadding="0"> |
| <tbody> |
| <tr id="projectrow"> |
| <td id="projectalign"> |
| <div id="projectname">iceberg-cpp |
| </div> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| </div> |
| <!-- end header part --> |
| <!-- Generated by Doxygen 1.9.8 --> |
| <script type="text/javascript"> |
| /* @license magnet:?xt=urn:btih:d3d9a9a6595521f9666a5e94cc830dab83b65699&dn=expat.txt MIT */ |
| var searchBox = new SearchBox("searchBox", "search/",'.html'); |
| /* @license-end */ |
| </script> |
| <script type="text/javascript" src="menudata.js"></script> |
| <script type="text/javascript" src="menu.js"></script> |
| <script type="text/javascript"> |
| /* @license magnet:?xt=urn:btih:d3d9a9a6595521f9666a5e94cc830dab83b65699&dn=expat.txt MIT */ |
| $(function() { |
| initMenu('',true,false,'search.php','Search'); |
| $(document).ready(function() { init_search(); }); |
| }); |
| /* @license-end */ |
| </script> |
| <div id="main-nav"></div> |
| <!-- window showing the filter options --> |
| <div id="MSearchSelectWindow" |
| onmouseover="return searchBox.OnSearchSelectShow()" |
| onmouseout="return searchBox.OnSearchSelectHide()" |
| onkeydown="return searchBox.OnSearchSelectKey(event)"> |
| </div> |
| |
| <!-- iframe showing the search results (closed by default) --> |
| <div id="MSearchResultsWindow"> |
| <div id="MSearchResults"> |
| <div class="SRPage"> |
| <div id="SRIndex"> |
| <div id="SRResults"></div> |
| <div class="SRStatus" id="Loading">Loading...</div> |
| <div class="SRStatus" id="Searching">Searching...</div> |
| <div class="SRStatus" id="NoMatches">No Matches</div> |
| </div> |
| </div> |
| </div> |
| </div> |
| |
| </div><!-- top --> |
| <div><div class="header"> |
| <div class="headertitle"><div class="title">README</div></div> |
| </div><!--header--> |
| <div class="contents"> |
| <div class="textblock"><h1><a class="anchor" id="autotoc_md0"></a> |
| SQL Catalog</h1> |
| <p><code>SqlCatalog</code> implements the Iceberg <code>Catalog</code> API on top of a relational database. Its on-disk schema is compatible with the Apache Iceberg Java <code>JdbcCatalog</code>: two tables, <code>iceberg_tables</code> and <code>iceberg_namespace_properties</code>, scoped by a catalog name so multiple catalogs can share one database.</p> |
| <h2><a class="anchor" id="autotoc_md1"></a> |
| Design</h2> |
| <p><code>SqlCatalog</code> owns the Iceberg catalog behavior. It validates namespaces, reads and writes table metadata files, and performs optimistic-concurrency commits. Database access is delegated to a small storage interface:</p> |
| <div class="fragment"><div class="line">Application</div> |
| <div class="line"> |</div> |
| <div class="line"> v</div> |
| <div class="line">SqlCatalog</div> |
| <div class="line"> |</div> |
| <div class="line"> | CatalogStore API</div> |
| <div class="line"> v</div> |
| <div class="line">CatalogStore implementation</div> |
| <div class="line"> |</div> |
| <div class="line"> v</div> |
| <div class="line">SQL database</div> |
| <div class="line"> - iceberg_tables</div> |
| <div class="line"> - iceberg_namespace_properties</div> |
| </div><!-- fragment --><p><code>CatalogStore</code> (see <a href="catalog_store.h"><code>catalog_store.h</code></a>) exposes typed row operations such as <code>InsertTable</code>, <code>GetTableMetadataLocation</code>, <code>UpdateTableMetadataLocation(expected_current)</code>, namespace-property CRUD, and <code>RunInTransaction</code>. It exposes no SQL strings or driver-specific types.</p> |
| <p>The project provides built-in <code>CatalogStore</code> implementations for SQLite, PostgreSQL, and MySQL. They are implemented with <a href="https://github.com/rbock/sqlpp23">sqlpp23</a>, and the shared query code lives in <code>catalog_store_sqlpp23_internal.h</code>. Users can also provide their own <code>CatalogStore</code> implementation for another database, driver, or connection pool.</p> |
| <p>sqlpp23 is a build-time-only dependency for the built-in stores. It is compiled into the connector translation units and does not appear in the installed interface, so downstream consumers only need the native client libraries.</p> |
| <blockquote class="doxtable"> |
| <p>‍The built-in sqlpp23 connectors require CMake >= 3.28 and C++23; sqlpp23 is fetched automatically via <code>FetchContent</code> when at least one built-in connector is enabled. A SQL catalog backed only by a user-supplied <code>CatalogStore</code> does not need sqlpp23. The SQL catalog is currently wired into the CMake build only; the Meson build does not build or install it yet. </p> |
| </blockquote> |
| <h2><a class="anchor" id="autotoc_md2"></a> |
| Out-of-the-box usage</h2> |
| <p>Enable the SQL catalog and any built-in connectors at configure time. Built-in connectors pull in their native client libraries via sqlpp23:</p> |
| <table class="markdownTable"> |
| <tr class="markdownTableHead"> |
| <th class="markdownTableHeadNone">CMake option </th><th class="markdownTableHeadNone">Default </th><th class="markdownTableHeadNone">sqlpp23 target </th><th class="markdownTableHeadNone">Native dependency </th></tr> |
| <tr class="markdownTableRowOdd"> |
| <td class="markdownTableBodyNone"><code>ICEBERG_BUILD_SQL_CATALOG</code> </td><td class="markdownTableBodyNone"><code>OFF</code> </td><td class="markdownTableBodyNone">- </td><td class="markdownTableBodyNone">- </td></tr> |
| <tr class="markdownTableRowEven"> |
| <td class="markdownTableBodyNone"><code>ICEBERG_SQL_SQLITE</code> </td><td class="markdownTableBodyNone"><code>OFF</code> </td><td class="markdownTableBodyNone"><code>sqlpp23::sqlite3</code> </td><td class="markdownTableBodyNone">SQLite3 </td></tr> |
| <tr class="markdownTableRowOdd"> |
| <td class="markdownTableBodyNone"><code>ICEBERG_SQL_POSTGRESQL</code> </td><td class="markdownTableBodyNone"><code>OFF</code> </td><td class="markdownTableBodyNone"><code>sqlpp23::postgresql</code> </td><td class="markdownTableBodyNone">libpq (PostgreSQL) </td></tr> |
| <tr class="markdownTableRowEven"> |
| <td class="markdownTableBodyNone"><code>ICEBERG_SQL_MYSQL</code> </td><td class="markdownTableBodyNone"><code>OFF</code> </td><td class="markdownTableBodyNone"><code>sqlpp23::mysql</code> </td><td class="markdownTableBodyNone">libmysqlclient (MySQL) </td></tr> |
| </table> |
| <div class="fragment"><div class="line">cmake -S . -B build -DICEBERG_BUILD_SQL_CATALOG=ON -DICEBERG_SQL_SQLITE=ON</div> |
| </div><!-- fragment --><div class="fragment"><div class="line"><span class="preprocessor">#include "<a class="code" href="sql__catalog_8h.html">iceberg/catalog/sql/sql_catalog.h</a>"</span></div> |
| <div class="line"> </div> |
| <div class="line"><span class="keyword">using </span><a class="code hl_class" href="classiceberg_1_1sql_1_1SqlCatalog.html">iceberg::sql::SqlCatalog</a>;</div> |
| <div class="line"><span class="keyword">using </span><a class="code hl_struct" href="structiceberg_1_1sql_1_1SqlCatalogConfig.html">iceberg::sql::SqlCatalogConfig</a>;</div> |
| <div class="line"> </div> |
| <div class="line">SqlCatalogConfig config{</div> |
| <div class="line"> .<a class="code hl_variable" href="structiceberg_1_1sql_1_1SqlCatalogConfig.html#a8b1dad5745a2992d54dbdda57bbcdc8d">name</a> = <span class="stringliteral">"prod"</span>,</div> |
| <div class="line"> .uri = <span class="stringliteral">"/var/lib/iceberg/catalog.db"</span>, <span class="comment">// SQLite file path</span></div> |
| <div class="line"> .warehouse_location = <span class="stringliteral">"s3://my-bucket/warehouse"</span>,</div> |
| <div class="line">};</div> |
| <div class="line"> </div> |
| <div class="line"><span class="keyword">auto</span> catalog = SqlCatalog::MakeSqliteCatalog(config, file_io).value();</div> |
| <div class="line"><span class="comment">// catalog->CreateNamespace(...), CreateTable(...), LoadTable(...), ...</span></div> |
| <div class="ttc" id="aclassiceberg_1_1sql_1_1SqlCatalog_html"><div class="ttname"><a href="classiceberg_1_1sql_1_1SqlCatalog.html">iceberg::sql::SqlCatalog</a></div><div class="ttdoc">SQL-backed Iceberg catalog.</div><div class="ttdef"><b>Definition</b> sql_catalog.h:67</div></div> |
| <div class="ttc" id="asql__catalog_8h_html"><div class="ttname"><a href="sql__catalog_8h.html">sql_catalog.h</a></div></div> |
| <div class="ttc" id="astructiceberg_1_1sql_1_1SqlCatalogConfig_html"><div class="ttname"><a href="structiceberg_1_1sql_1_1SqlCatalogConfig.html">iceberg::sql::SqlCatalogConfig</a></div><div class="ttdoc">Configuration for the SQL catalog.</div><div class="ttdef"><b>Definition</b> sql_catalog.h:49</div></div> |
| <div class="ttc" id="astructiceberg_1_1sql_1_1SqlCatalogConfig_html_a8b1dad5745a2992d54dbdda57bbcdc8d"><div class="ttname"><a href="structiceberg_1_1sql_1_1SqlCatalogConfig.html#a8b1dad5745a2992d54dbdda57bbcdc8d">iceberg::sql::SqlCatalogConfig::name</a></div><div class="ttdeci">std::string name</div><div class="ttdef"><b>Definition</b> sql_catalog.h:52</div></div> |
| </div><!-- fragment --><p><code>MakePostgreSqlCatalog</code> and <code>MakeMySqlCatalog</code> use the same <code>[scheme://][user[:password]@]host[:port][/database]</code> URI form. Connector factories are always declared in the public headers; if a connector was not built, its factory returns <code>ErrorKind::kNotSupported</code>. Each enabled factory creates the schema if it does not yet exist.</p> |
| <p>The PostgreSQL and MySQL stores use a single sqlpp23 connection when <code>max_connections <= 1</code> and a bounded sqlpp23 connection pool otherwise. Transaction bodies reuse the same leased connection for every store operation issued inside <code>RunInTransaction</code>. The SQLite store ignores <code>max_connections</code> and always uses a single connection: a file database only allows one writer (a pool of write connections would just hit <code>SQLITE_BUSY</code>) and a <code>:memory:</code> database is private to each connection.</p> |
| <p>The backing schema follows the Java/Rust-compatible <code>iceberg_tables</code> layout, including the optional <code>iceberg_type</code> column. New table rows write ‘iceberg_type = 'TABLE’<code>as the record type; existing rows with</code>NULL` remain readable for compatibility.</p> |
| <h2><a class="anchor" id="autotoc_md3"></a> |
| Bring your own store</h2> |
| <p>To use a database, driver, or connection pool that is not built in, implement <code>CatalogStore</code> and inject it. No catalog code changes are required:</p> |
| <div class="fragment"><div class="line"><span class="keyword">class </span>MyCatalogStore : <span class="keyword">public</span> <a class="code hl_class" href="classiceberg_1_1sql_1_1CatalogStore.html">iceberg::sql::CatalogStore</a> {</div> |
| <div class="line"> <span class="keyword">public</span>:</div> |
| <div class="line"> iceberg::Status <a class="code hl_function" href="classiceberg_1_1sql_1_1CatalogStore.html#ae56f64bd20de171ac60a6b5540f4f7a9">Initialize</a>()<span class="keyword"> override </span>{ <span class="comment">/* CREATE TABLE IF NOT EXISTS ... */</span> }</div> |
| <div class="line"> <a class="code hl_typedef" href="namespaceiceberg.html#a9d5594e371beb516b0120e07e69d3af6">iceberg::Result<std::optional<std::string></a>> <a class="code hl_function" href="classiceberg_1_1sql_1_1CatalogStore.html#a047c2b1353469bfdda8739b846fbde46">GetTableMetadataLocation</a>(</div> |
| <div class="line"> std::string_view ns, std::string_view name)<span class="keyword"> override </span>{ <span class="comment">/* ... */</span> }</div> |
| <div class="line"> iceberg::Status <a class="code hl_function" href="classiceberg_1_1sql_1_1CatalogStore.html#a393d3119d8b17efa36f7021189fd02e4">InsertTable</a>(std::string_view ns, std::string_view name,</div> |
| <div class="line"> std::string_view metadata_location)<span class="keyword"> override </span>{ <span class="comment">/* ... */</span> }</div> |
| <div class="line"> <span class="comment">// ... the remaining CatalogStore operations ...</span></div> |
| <div class="line"> iceberg::Status <a class="code hl_function" href="classiceberg_1_1sql_1_1CatalogStore.html#aa8662070d94c72e3671367a726f7427e">RunInTransaction</a>(</div> |
| <div class="line"> <span class="keyword">const</span> std::function<iceberg::Status()>& body)<span class="keyword"> override </span>{ <span class="comment">/* ... */</span> }</div> |
| <div class="line">};</div> |
| <div class="line"> </div> |
| <div class="line"><span class="keyword">auto</span> store = std::make_shared<MyCatalogStore>(<span class="comment">/* ... */</span>);</div> |
| <div class="line"><span class="keyword">auto</span> catalog = SqlCatalog::Make(config, file_io, std::move(store)).value();</div> |
| <div class="ttc" id="aclassiceberg_1_1sql_1_1CatalogStore_html"><div class="ttname"><a href="classiceberg_1_1sql_1_1CatalogStore.html">iceberg::sql::CatalogStore</a></div><div class="ttdoc">Semantic, driver-agnostic storage interface for the SQL catalog.</div><div class="ttdef"><b>Definition</b> catalog_store.h:72</div></div> |
| <div class="ttc" id="aclassiceberg_1_1sql_1_1CatalogStore_html_a047c2b1353469bfdda8739b846fbde46"><div class="ttname"><a href="classiceberg_1_1sql_1_1CatalogStore.html#a047c2b1353469bfdda8739b846fbde46">iceberg::sql::CatalogStore::GetTableMetadataLocation</a></div><div class="ttdeci">virtual Result< std::optional< std::string > > GetTableMetadataLocation(std::string_view ns, std::string_view name)=0</div><div class="ttdoc">Return the current metadata location of (ns, name).</div></div> |
| <div class="ttc" id="aclassiceberg_1_1sql_1_1CatalogStore_html_a393d3119d8b17efa36f7021189fd02e4"><div class="ttname"><a href="classiceberg_1_1sql_1_1CatalogStore.html#a393d3119d8b17efa36f7021189fd02e4">iceberg::sql::CatalogStore::InsertTable</a></div><div class="ttdeci">virtual Status InsertTable(std::string_view ns, std::string_view name, std::string_view metadata_location)=0</div><div class="ttdoc">Insert a new table row with the given metadata location.</div></div> |
| <div class="ttc" id="aclassiceberg_1_1sql_1_1CatalogStore_html_aa8662070d94c72e3671367a726f7427e"><div class="ttname"><a href="classiceberg_1_1sql_1_1CatalogStore.html#aa8662070d94c72e3671367a726f7427e">iceberg::sql::CatalogStore::RunInTransaction</a></div><div class="ttdeci">virtual Status RunInTransaction(const std::function< Status()> &body)=0</div><div class="ttdoc">Execute body atomically inside a single transaction.</div></div> |
| <div class="ttc" id="aclassiceberg_1_1sql_1_1CatalogStore_html_ae56f64bd20de171ac60a6b5540f4f7a9"><div class="ttname"><a href="classiceberg_1_1sql_1_1CatalogStore.html#ae56f64bd20de171ac60a6b5540f4f7a9">iceberg::sql::CatalogStore::Initialize</a></div><div class="ttdeci">virtual Status Initialize()=0</div><div class="ttdoc">Create the backing tables if they do not already exist.</div></div> |
| <div class="ttc" id="anamespaceiceberg_html_a9d5594e371beb516b0120e07e69d3af6"><div class="ttname"><a href="namespaceiceberg.html#a9d5594e371beb516b0120e07e69d3af6">iceberg::Result</a></div><div class="ttdeci">std::expected< T, E > Result</div><div class="ttdoc">Result alias.</div><div class="ttdef"><b>Definition</b> result.h:88</div></div> |
| </div><!-- fragment --><h3><a class="anchor" id="autotoc_md4"></a> |
| Implementation contract</h3> |
| <ul> |
| <li><b>Catalog scope</b>: a store instance is bound to one catalog name; every row it reads or writes must be scoped by that name.</li> |
| <li><b>Namespace identifiers</b>: namespace levels must not be empty and must not contain <code>.</code> because the backing schema stores a namespace as a dot-joined string.</li> |
| <li><b>Table rows</b>: new table inserts should write ‘iceberg_type = 'TABLE’<code>as the record type. Reads should treat both</code>TABLE<code>and</code>NULL<code>as table rows so older databases remain readable.</code></li> |
| <li><code>**Unique violations**:</code>InsertTable<code>,</code>InsertNamespaceProperty<code>, and </code>RenameTable<code>must report a primary-key collision as</code>ErrorKind::kAlreadyExists<code>. The catalog relies on this as the authoritative signal for concurrent creates.</code></li> |
| <li><code>**Affected rows**:</code>UpdateTableMetadataLocation<code>performs the optimistic compare-and-set; it must return the number of rows updated (0 on a stale base).</code></li> |
| <li><code>**Atomicity**:</code>RunInTransaction<code>must commit on success and roll back on any error so the database is left unchanged.</code></li> |
| <li><code>**Threading**: a store may be called from multiple threads; serialize internally or use one connection per concurrent operation. The built-in PostgreSQL and MySQL stores use a bounded sqlpp23 connection pool when </code>max_connections > 1`; the SQLite store always uses a single connection. </li> |
| </ul> |
| </div></div><!-- contents --> |
| </div><!-- PageDoc --> |
| <!-- start footer part --> |
| <hr class="footer"/><address class="footer"><small> |
| Generated by <a href="https://www.doxygen.org/index.html"><img class="footer" src="doxygen.svg" width="104" height="31" alt="doxygen"/></a> 1.9.8 |
| </small></address> |
| </body> |
| </html> |
