Google Git
Sign in
apache / jena-site / refs/heads/asf-site / . / content / documentation / tdb / store-parameters.html
blob: 2d94db1bddd8a9ada19b3d11df98286132717fee [file] [log] [blame]
<!DOCTYPE html>
<html lang="en">
<head>
<title>Apache Jena - TDB - Store Parameters</title>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<link href="/css/bootstrap.min.css" rel="stylesheet" media="screen">
<link href="/css/bootstrap-icons.css" rel="stylesheet" media="screen"><link rel="stylesheet" type="text/css" href="https://jena.apache.org/sass/jena.1b17c39a117e22b46db4c66f6395dc27c134a60377d87d2d5745b8600eb69722.css" integrity="sha256-GxfDmhF&#43;IrRttMZvY5XcJ8E0pgN32H0tV0W4YA62lyI=">
<link rel="shortcut icon" href="/images/favicon.ico" />
</head>
<body>
<nav class="navbar navbar-expand-lg bg-body-tertiary" role="navigation">
<div class="container">
<div class="navbar-header">
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarNav" aria-controls="navbarNav" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
<a class="navbar-brand" href="/index.html">
<img class="logo-menu" src="/images/jena-logo/jena-logo-notext-small.png" alt="jena logo">Apache Jena</a>
</div>
<div class="collapse navbar-collapse" id="navbarNav">
<ul class="navbar-nav me-auto mb-2 mb-lg-0">
<li id="homepage" class="nav-item"><a class="nav-link" href="/index.html"><span class="bi-house"></span> Home</a></li>
<li id="download" class="nav-item"><a class="nav-link" href="/download/index.cgi"><span class="bi-download"></span> Download</a></li>
<li class="nav-item dropdown">
<a href="#" class="nav-link dropdown-toggle" role="button" data-bs-toggle="dropdown" aria-expanded="false"><span class="bi-journal"></span> Learn <b class="caret"></b></a>
<ul class="dropdown-menu">
<li class="dropdown-header">Tutorials</li>
<li><a class="dropdown-item" href="/tutorials/index.html">Overview</a></li>
<li><a class="dropdown-item" href="/documentation/fuseki2/index.html">Fuseki Triplestore</a></li>
<li><a class="dropdown-item" href="/documentation/notes/index.html">How-To's</a></li>
<li><a class="dropdown-item" href="/documentation/query/manipulating_sparql_using_arq.html">Manipulating SPARQL using ARQ</a></li>
<li><a class="dropdown-item" href="/tutorials/rdf_api.html">RDF core API tutorial</a></li>
<li><a class="dropdown-item" href="/tutorials/sparql.html">SPARQL tutorial</a></li>
<li><a class="dropdown-item" href="/tutorials/using_jena_with_eclipse.html">Using Jena with Eclipse</a></li>
<li class="dropdown-divider"></li>
<li class="dropdown-header">References</li>
<li><a class="dropdown-item" href="/documentation/index.html">Overview</a></li>
<li><a class="dropdown-item" href="/documentation/query/index.html">ARQ (SPARQL)</a></li>
<li><a class="dropdown-item" href="/documentation/io/">RDF I/O</a></li>
<li><a class="dropdown-item" href="/documentation/assembler/index.html">Assembler</a></li>
<li><a class="dropdown-item" href="/documentation/tools/index.html">Command-line tools</a></li>
<li><a class="dropdown-item" href="/documentation/rdfs/">Data with RDFS Inferencing</a></li>
<li><a class="dropdown-item" href="/documentation/geosparql/index.html">GeoSPARQL</a></li>
<li><a class="dropdown-item" href="/documentation/inference/index.html">Inference API</a></li>
<li><a class="dropdown-item" href="/documentation/ontology/">Ontology API</a></li>
<li><a class="dropdown-item" href="/documentation/permissions/index.html">Permissions</a></li>
<li><a class="dropdown-item" href="/documentation/extras/querybuilder/index.html">Query Builder</a></li>
<li><a class="dropdown-item" href="/documentation/rdf/index.html">RDF API</a></li>
<li><a class="dropdown-item" href="/documentation/rdfconnection/">RDF Connection - SPARQL API</a></li>
<li><a class="dropdown-item" href="/documentation/rdfstar/index.html">RDF-star</a></li>
<li><a class="dropdown-item" href="/documentation/shacl/index.html">SHACL</a></li>
<li><a class="dropdown-item" href="/documentation/shex/index.html">ShEx</a></li>
<li><a class="dropdown-item" href="/documentation/tdb/index.html">TDB</a></li>
<li><a class="dropdown-item" href="/documentation/tdb2/index.html">TDB2</a></li>
<li><a class="dropdown-item" href="/documentation/query/text-query.html">Text Search</a></li>
</ul>
</li>
<li class="nav-item dropdown">
<a href="#" class="nav-link dropdown-toggle" role="button" data-bs-toggle="dropdown" aria-expanded="false"><span class="bi-journal-code"></span> Javadoc <b class="caret"></b></a>
<ul class="dropdown-menu">
<li><a class="dropdown-item" href="/documentation/javadoc.html">All Javadoc</a></li>
<li><a class="dropdown-item" href="/documentation/javadoc/arq/">ARQ</a></li>
<li><a class="dropdown-item" href="/documentation/javadoc/fuseki2/">Fuseki</a></li>
<li><a class="dropdown-item" href="/documentation/javadoc/geosparql/">GeoSPARQL</a></li>
<li><a class="dropdown-item" href="/documentation/javadoc/jena/">Jena Core</a></li>
<li><a class="dropdown-item" href="/documentation/javadoc/permissions/">Permissions</a></li>
<li><a class="dropdown-item" href="/documentation/javadoc/extras/querybuilder/">Query Builder</a></li>
<li><a class="dropdown-item" href="/documentation/javadoc/shacl/">SHACL</a></li>
<li><a class="dropdown-item" href="/documentation/javadoc/tdb/">TDB</a></li>
<li><a class="dropdown-item" href="/documentation/javadoc/text/">Text Search</a></li>
</ul>
</li>
</ul>
<form class="d-flex" role="search" action="/search" method="GET">
<div class="input-group">
<input class="form-control border-end-0 border m-0" type="search" name="q" id="search-query" placeholder="Search...." aria-label="Search" style="width: 10rem;">
<button class="btn btn-outline-secondary border-start-0 border" type="submit">
<i class="bi-search"></i>
</button>
</div>
</form>
<ul class="navbar-nav">
<li id="ask" class="nav-item"><a class="nav-link" href="/help_and_support/index.html" title="Ask"><span class="bi-patch-question"></span><span class="text-body d-none d-xxl-inline"> Ask</span></a></li>
<li class="nav-item dropdown">
<a href="#" title="Get involved" class="nav-link dropdown-toggle" role="button" data-bs-toggle="dropdown" aria-expanded="false"><span class="bi-megaphone"></span><span class="text-body d-none d-xxl-inline"> Get involved </span><b class="caret"></b></a>
<ul class="dropdown-menu">
<li><a class="dropdown-item" href="/getting_involved/index.html">Contribute</a></li>
<li><a class="dropdown-item" href="/help_and_support/bugs_and_suggestions.html">Report a bug</a></li>
<li class="dropdown-divider"></li>
<li class="dropdown-header">Project</li>
<li><a class="dropdown-item" href="/about_jena/about.html">About Jena</a></li>
<li><a class="dropdown-item" href="/about_jena/architecture.html">Architecture</a></li>
<li><a class="dropdown-item" href="/about_jena/citing.html">Citing</a></li>
<li><a class="dropdown-item" href="/about_jena/team.html">Project team</a></li>
<li><a class="dropdown-item" href="/about_jena/contributions.html">Related projects</a></li>
<li><a class="dropdown-item" href="/about_jena/roadmap.html">Roadmap</a></li>
<li><a class="dropdown-item" href="/about_jena/security-advisories.html">Security Advisories</a></li>
<li class="dropdown-divider"></li>
<li class="dropdown-header">ASF</li>
<li><a class="dropdown-item" href="https://www.apache.org/">Apache Software Foundation</a></li>
<li><a class="dropdown-item" href="https://www.apache.org/foundation/sponsorship.html">Become a Sponsor</a></li>
<li><a class="dropdown-item" href="https://www.apache.org/licenses/LICENSE-2.0">License</a></li>
<li><a class="dropdown-item" href="https://www.apache.org/security/">Security</a></li>
<li><a class="dropdown-item" href="https://www.apache.org/foundation/thanks.html">Thanks</a></li>
</ul>
</li>
<li class="nav-item" id="edit"><a class="nav-link" href="https://github.com/apache/jena-site/edit/main/source/documentation/tdb/store-parameters.md" title="Edit this page on GitHub"><span class="bi-pencil-square"></span><span class="text-body d-none d-xxl-inline"> Edit this page</span></a></li>
</ul>
</div>
</div>
</nav>
<div class="container">
<div class="row">
<div class="col-md-12">
<div id="breadcrumbs">
<ol class="breadcrumb mt-4 p-2 bg-body-tertiary">
<li class="breadcrumb-item"><a href='/documentation'>DOCUMENTATION</a></li>
<li class="breadcrumb-item"><a href='/documentation/tdb'>TDB</a></li>
<li class="breadcrumb-item active">STORE PARAMETERS</li>
</ol>
</div>
<h1 class="title">TDB - Store Parameters</h1>
<main class="d-flex flex-xl-row flex-column">
<aside class="text-muted align-self-start mb-3 p-0 d-xl-none d-block">
<h2 class="h6 sticky-top m-0 p-2 bg-body-tertiary">On this page</h2>
<nav id="TableOfContents">
<ul>
<li><a href="#setting-store-parameters">Setting Store Parameters</a></li>
<li><a href="#per-connect-options">Per-connect Options</a>
<ul>
<li><a href="#file-access---mapped-and-direct-modes">File access - &ldquo;mapped&rdquo; and &ldquo;direct&rdquo; modes</a></li>
<li><a href="#caching-options">Caching options.</a></li>
</ul>
</li>
<li><a href="#static-options">Static Options</a>
<ul>
<li><a href="#block-size">Block Size</a></li>
</ul>
</li>
<li><a href="#store-parameters-file-format">Store Parameters File Format</a></li>
<li><a href="#choosing-the-store-parameters">Choosing the store parameters</a></li>
</ul>
</nav>
</aside>
<article class="flex-column me-lg-4">
<p>TDB (as of version Jena 3.0.0) supports configuration of the
databases when they are first created and each time an application connects
to an existing database. Databases using the default settings built-into
TDB continue to work exactly as before.</p>
<h2 id="setting-store-parameters">Setting Store Parameters</h2>
<p>In TDB, there is exactly one internal object for each dataset in the JVM
and this is shared between all application datasets for that location of
persistent storage.</p>
<p>Setting store parameters is done by setting the internal system state
before any other access to the disk area occurs. It is not possible to have
different setups for the same dataset on disk.</p>
<p><code>StoreParams</code> are set by populating the internal state with the setup
before an application level dataset is created.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-java" data-lang="java"><span style="display:flex;"><span>TDBFactory<span style="color:#666">.</span><span style="color:#b44">setup</span><span style="color:#666">(</span>Location location<span style="color:#666">,</span> StoreParams params<span style="color:#666">)</span>
</span></span></code></pre></div><p>This must be called before any application calls to get a <code>Dataset</code> (or
<code>DatasetGraph</code>) object otherwise <code>IllegalStateException</code> is thrown by this
function.</p>
<pre tabindex="0"><code class="language-hava" data-lang="hava">Location location = ... ;
StoreParams customParams = ... ;
TDBFactory.setup(location, customParams) ;
Dataset ds = TDBFactory.createDataset(location) ;
...
</code></pre><p>It is only possible to change store parameters by expelling the managed
storage by calling <code>TDBFactory.release(Location)</code>. This drops all caching.
Access to the dataset is then a cold start.</p>
<h2 id="per-connect-options">Per-connect Options</h2>
<p>The per-connect options are the ones that can be changed after the database has
been created and can be different each time the application attaches to the
database. A database can have at most one JVM attached to it (see Fuseki
to share a database).</p>
<p>These options do not affect the on-disk structures.</p>
<table>
<thead>
<tr>
<th>JSON key name</th>
<th style="text-align:right">Default value</th>
<th>Notes</th>
</tr>
</thead>
<tbody>
<tr>
<td>tdb.file_mode</td>
<td style="text-align:right">See below</td>
<td></td>
</tr>
<tr>
<td>tdb.node2nodeid_cache_size</td>
<td style="text-align:right">100,000</td>
<td>50,000 on 32 bit java</td>
</tr>
<tr>
<td>tdb.nodeid2node_cache_size</td>
<td style="text-align:right">500,000</td>
<td>50,000 on 32 bit java</td>
</tr>
<tr>
<td>tdb.node_miss_cache_size</td>
<td style="text-align:right">100</td>
<td></td>
</tr>
<tr>
<td>tdb.block_read_cache_size</td>
<td style="text-align:right">10000</td>
<td>Only in direct mode</td>
</tr>
<tr>
<td>tdb.block_write_cache_size</td>
<td style="text-align:right">2000</td>
<td>Only in direct mode</td>
</tr>
</tbody>
</table>
<h3 id="file-access---mapped-and-direct-modes">File access - &ldquo;mapped&rdquo; and &ldquo;direct&rdquo; modes</h3>
<p>TDB has two modes of operation for accessing block files - &ldquo;mapped&rdquo; and
&ldquo;direct&rdquo;.</p>
<ul>
<li>
<p>&ldquo;mapped&rdquo; uses memory mapped files and so the operating system is managing
caching, flexing the amount of memory for file system cache to balance
demands from other programmes on the same hardware.</p>
</li>
<li>
<p>&ldquo;direct&rdquo; using TDB&rsquo;s own in-heap block caching. It avoids the problem that
addressing is limited to a total of about 1.5Gbytes on 32 bit Java.</p>
</li>
</ul>
<p>By default, TDB uses memory mapped files on 64 bit Java and its own file
caching on 32 bit java.</p>
<p>On Microsoft Windows, &ldquo;mapped&rdquo; databases can not be deleted while the JVM is running on MS
Windows. This is a <a href="http://bugs.java.com/view_bug.do?bug_id=4715154">known issue with Java</a>.</p>
<p>TDB databases are compatible across these file modes. There is no
difference to the file layouts. Memory mapped files may appear larger
because they contain unused space. Some utilities report this in file
size, some do not.</p>
<h3 id="caching-options">Caching options.</h3>
<p>These are the useful tuning options. Only the <tt>node*</tt> choices have
any effect when running in &ldquo;mapped&rdquo; mode.</p>
<p>All these options effect the amount of heap used. The block read/write
cache sizes are tuned to 32 bit Java.</p>
<p>Increasing the Node/NodeId cache sizes on 64 bit machines may be
beneficial.</p>
<h2 id="static-options">Static Options</h2>
<p>While it is possible to customize a database, this is considered to be
experimental. It is possible to corrupt, unrecoverable, existing databases
and create nonsense databases with inappropriate settings. It will be
useful in very few real situations. Not all combinations of index choices
will work. Only the standard layout is supported; alternative schemes are
for experimentation only.</p>
<h3 id="block-size">Block Size</h3>
<p>The block size can not be changed once a database has been created.</p>
<p>While the code attempts to detect block size mismatches, in order to retain
compatibility with existing database, the testing can not be perfect. If
undetected, any update will permanently and irrecoverably damage the
database.</p>
<h2 id="store-parameters-file-format">Store Parameters File Format</h2>
<p>JSON is used for the on-disk record of store parameters, see the example
below. Unspecified options defaults to the for the running setup.</p>
<p>These are default settings for a 64 bit Java:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-json" data-lang="json"><span style="display:flex;"><span>{
</span></span><span style="display:flex;"><span> <span style="color:#008000;font-weight:bold">&#34;tdb.file_mode&#34;</span> : <span style="color:#b44">&#34;mapped&#34;</span> ,
</span></span><span style="display:flex;"><span> <span style="color:#008000;font-weight:bold">&#34;tdb.block_size&#34;</span> : <span style="color:#666">8192</span> ,
</span></span><span style="display:flex;"><span> <span style="color:#008000;font-weight:bold">&#34;tdb.block_read_cache_size&#34;</span> : <span style="color:#666">10000</span> ,
</span></span><span style="display:flex;"><span> <span style="color:#008000;font-weight:bold">&#34;tdb.block_write_cache_size&#34;</span> : <span style="color:#666">2000</span> ,
</span></span><span style="display:flex;"><span> <span style="color:#008000;font-weight:bold">&#34;tdb.node2nodeid_cache_size&#34;</span> : <span style="color:#666">100000</span> ,
</span></span><span style="display:flex;"><span> <span style="color:#008000;font-weight:bold">&#34;tdb.nodeid2node_cache_size&#34;</span> : <span style="color:#666">500000</span> ,
</span></span><span style="display:flex;"><span> <span style="color:#008000;font-weight:bold">&#34;tdb.node_miss_cache_size&#34;</span> : <span style="color:#666">100</span> ,
</span></span><span style="display:flex;"><span> <span style="color:#008000;font-weight:bold">&#34;tdb.index_node2id&#34;</span> : <span style="color:#b44">&#34;node2id&#34;</span> ,
</span></span><span style="display:flex;"><span> <span style="color:#008000;font-weight:bold">&#34;tdb.index_id2node&#34;</span> : <span style="color:#b44">&#34;nodes&#34;</span> ,
</span></span><span style="display:flex;"><span> <span style="color:#008000;font-weight:bold">&#34;tdb.triple_index_primary&#34;</span> : <span style="color:#b44">&#34;SPO&#34;</span> ,
</span></span><span style="display:flex;"><span> <span style="color:#008000;font-weight:bold">&#34;tdb.triple_indexes&#34;</span> : [ <span style="color:#b44">&#34;SPO&#34;</span> , <span style="color:#b44">&#34;POS&#34;</span> , <span style="color:#b44">&#34;OSP&#34;</span> ] ,
</span></span><span style="display:flex;"><span> <span style="color:#008000;font-weight:bold">&#34;tdb.quad_index_primary&#34;</span> : <span style="color:#b44">&#34;GSPO&#34;</span> ,
</span></span><span style="display:flex;"><span> <span style="color:#008000;font-weight:bold">&#34;tdb.quad_indexes&#34;</span> : [ <span style="color:#b44">&#34;GSPO&#34;</span> , <span style="color:#b44">&#34;GPOS&#34;</span> , <span style="color:#b44">&#34;GOSP&#34;</span> , <span style="color:#b44">&#34;POSG&#34;</span> , <span style="color:#b44">&#34;OSPG&#34;</span> , <span style="color:#b44">&#34;SPOG&#34;</span> ] ,
</span></span><span style="display:flex;"><span> <span style="color:#008000;font-weight:bold">&#34;tdb.prefix_index_primary&#34;</span> : <span style="color:#b44">&#34;GPU&#34;</span> ,
</span></span><span style="display:flex;"><span> <span style="color:#008000;font-weight:bold">&#34;tdb.prefix_indexes&#34;</span> : [ <span style="color:#b44">&#34;GPU&#34;</span> ] ,
</span></span><span style="display:flex;"><span> <span style="color:#008000;font-weight:bold">&#34;tdb.file_prefix_index&#34;</span> : <span style="color:#b44">&#34;prefixIdx&#34;</span> ,
</span></span><span style="display:flex;"><span> <span style="color:#008000;font-weight:bold">&#34;tdb.file_prefix_nodeid&#34;</span> : <span style="color:#b44">&#34;prefix2id&#34;</span> ,
</span></span><span style="display:flex;"><span> <span style="color:#008000;font-weight:bold">&#34;tdb.file_prefix_id2node&#34;</span> : <span style="color:#b44">&#34;prefixes&#34;</span>
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><h2 id="choosing-the-store-parameters">Choosing the store parameters</h2>
<p>This is the policy applied when creating or reattaching to a database.</p>
<p>If the database location has a parameter file, <code>tdb.cfg</code> then use that.
This is modified by any dynamic options supplied by the application. So to
create a specialized database, one way to do that is to create an empty
directory and put a <code>tdb.cfg</code> in place.</p>
<p>If there is no parameter file and this is a new database, use the
application provided store parameters, or if there are no application
provided parameters, use the system default parameters. If application
supplied parameters are used, write a <code>tdb.cfg</code> file.</p>
<p>Finally, if this is an existing database, with no <code>tdb.cfg</code>, use the system
default modified by any application parameters.</p>
<p>In other words, if there is no <code>tdb.cfg</code> assume the system defaults, except
when creating a database.</p>
<p><em>Modification</em> involves taking one set of store parameters and applying any
dynamic parameters set in the second set. Only explicitly set dynamic
parameters modify the original.</p>
</article>
<aside class="text-muted align-self-start mb-3 mb-xl-5 p-0 d-none d-xl-flex flex-column sticky-top">
<h2 class="h6 sticky-top m-0 p-2 bg-body-tertiary">On this page</h2>
<nav id="TableOfContents">
<ul>
<li><a href="#setting-store-parameters">Setting Store Parameters</a></li>
<li><a href="#per-connect-options">Per-connect Options</a>
<ul>
<li><a href="#file-access---mapped-and-direct-modes">File access - &ldquo;mapped&rdquo; and &ldquo;direct&rdquo; modes</a></li>
<li><a href="#caching-options">Caching options.</a></li>
</ul>
</li>
<li><a href="#static-options">Static Options</a>
<ul>
<li><a href="#block-size">Block Size</a></li>
</ul>
</li>
<li><a href="#store-parameters-file-format">Store Parameters File Format</a></li>
<li><a href="#choosing-the-store-parameters">Choosing the store parameters</a></li>
</ul>
</nav>
</aside>
</main>
</div>
</div>
</div>
<footer class="bd-footer py-4 py-md-5 mt-4 mt-lg-5 bg-body-tertiary">
<div class="container" style="font-size:80%" >
<p>
Copyright &copy; 2011&ndash;2024 The Apache Software Foundation, Licensed under the
<a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.
</p>
<p>
Apache Jena, Jena, the Apache Jena project logo, Apache and the Apache feather logos are trademarks of
The Apache Software Foundation.
<br/>
<a href="https://privacy.apache.org/policies/privacy-policy-public.html"
>Apache Software Foundation Privacy Policy</a>.
</p>
</div>
</footer>
<script src="/js/popper.min.js.js" type="text/javascript"></script>
<script src="/js/bootstrap.min.js" type="text/javascript"></script>
<script src="/js/improve.js" type="text/javascript"></script>
<script type="text/javascript">
(function() {
'use strict'
const links = document.querySelectorAll(`a[href="${window.location.pathname}"]`)
if (links !== undefined && links !== null) {
for (const link of links) {
link.classList.add('active')
let parentElement = link.parentElement
let count = 0
const levelsLimit = 4
while (['UL', 'LI'].includes(parentElement.tagName) && count <= levelsLimit) {
if (parentElement.tagName === 'LI') {
parentElement.querySelector('a:first-child').classList.add('active')
}
parentElement = parentElement.parentElement
count++
}
}
}
})()
</script>
</body>
</html>