| <!DOCTYPE html> |
| <!-- |
| | Generated by Apache Maven Doxia Site Renderer 1.8.1 at 2020-11-23 |
| | Rendered using Apache Maven Fluido Skin 1.5 |
| --> |
| <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> |
| <head> |
| <meta charset="UTF-8" /> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0" /> |
| <meta name="Date-Revision-yyyymmdd" content="20201123" /> |
| <meta http-equiv="Content-Language" content="en" /> |
| <title>Apache Phoenix Omid – Management of Client Failures</title> |
| <link rel="stylesheet" href="./css/apache-maven-fluido-1.5.min.css" /> |
| <link rel="stylesheet" href="./css/site.css" /> |
| <link rel="stylesheet" href="./css/print.css" media="print" /> |
| |
| |
| <script type="text/javascript" src="./js/apache-maven-fluido-1.5.min.js"></script> |
| |
| </head> |
| <body class="topBarEnabled"> |
| |
| |
| |
| |
| |
| <div id="topbar" class="navbar navbar-fixed-top "> |
| <div class="navbar-inner"> |
| <div class="container" style="width: 100%;"><div class="nav-collapse"> |
| |
| |
| <ul class="nav"> |
| <li class="dropdown"> |
| <a href="#" class="dropdown-toggle" data-toggle="dropdown">Home <b class="caret"></b></a> |
| <ul class="dropdown-menu"> |
| |
| <li> <a href="index.html" title="Overview">Overview</a> |
| </li> |
| |
| <li> <a href="licenses.html" title="Licenses">Licenses</a> |
| </li> |
| </ul> |
| </li> |
| <li class="dropdown"> |
| <a href="#" class="dropdown-toggle" data-toggle="dropdown">Download <b class="caret"></b></a> |
| <ul class="dropdown-menu"> |
| |
| <li> <a href="https://github.com/apache/phoenix-omid" title="Omid Source Repository">Omid Source Repository</a> |
| </li> |
| |
| <li> <a href="https://dist.apache.org/repos/dist/release/incubator/omid/" title="Apache Releases for version 1.0.1 and earlier">Apache Releases for version 1.0.1 and earlier</a> |
| </li> |
| |
| <li> <a href="https://dist.apache.org/repos/dist/release/phoenix/" title="Apache Releases for version 1.0.2 and later">Apache Releases for version 1.0.2 and later</a> |
| </li> |
| </ul> |
| </li> |
| <li class="dropdown"> |
| <a href="#" class="dropdown-toggle" data-toggle="dropdown">User Guide & API <b class="caret"></b></a> |
| <ul class="dropdown-menu"> |
| |
| <li> <a href="quickstart.html" title="Quickstart">Quickstart</a> |
| </li> |
| |
| <li> <a href="basic-examples.html" title="API and Code Examples">API and Code Examples</a> |
| </li> |
| </ul> |
| </li> |
| <li class="dropdown"> |
| <a href="#" class="dropdown-toggle" data-toggle="dropdown">Technical Docs <b class="caret"></b></a> |
| <ul class="dropdown-menu"> |
| |
| <li> <a href="basic-concepts.html" title="Basic Concepts">Basic Concepts</a> |
| </li> |
| |
| <li> <a href="omid-components.html" title="Omid Components">Omid Components</a> |
| </li> |
| |
| <li> <a href="basic-algorithm.html" title="Basic Algorithm">Basic Algorithm</a> |
| </li> |
| |
| <li> <a href="client-failure-management.html" title="Management of Client Failures">Management of Client Failures</a> |
| </li> |
| |
| <li> <a href="http://yahoohadoop.tumblr.com/tagged/HBase" title="Blog Entries">Blog Entries</a> |
| </li> |
| </ul> |
| </li> |
| <li class="dropdown"> |
| <a href="#" class="dropdown-toggle" data-toggle="dropdown">Contribute <b class="caret"></b></a> |
| <ul class="dropdown-menu"> |
| |
| <li> <a href="https://gitbox.apache.org/repos/asf/phoenix-omid.git" title="Source Code">Source Code</a> |
| </li> |
| |
| <li> <a href="https://issues.apache.org/jira/browse/Omid" title="JIRA">JIRA</a> |
| </li> |
| |
| <li> <a href="mailing-lists.html" title="Mailing Lists">Mailing Lists</a> |
| </li> |
| |
| <li> <a href="coding-guide-and-style.html" title="Coding Guide and Style">Coding Guide and Style</a> |
| </li> |
| </ul> |
| </li> |
| <li class="dropdown"> |
| <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Reports <b class="caret"></b></a> |
| <ul class="dropdown-menu"> |
| |
| <li class="dropdown-submenu"> |
| <a href="project-info.html" title="Project Information">Project Information</a> |
| <ul class="dropdown-menu"> |
| <li> <a href="dependency-convergence.html" title="Dependency Convergence">Dependency Convergence</a> |
| </li> |
| <li> <a href="dependency-info.html" title="Dependency Information">Dependency Information</a> |
| </li> |
| <li> <a href="dependency-management.html" title="Dependency Management">Dependency Management</a> |
| </li> |
| <li> <a href="distribution-management.html" title="Distribution Management">Distribution Management</a> |
| </li> |
| <li> <a href="index.html" title="About">About</a> |
| </li> |
| <li> <a href="licenses.html" title="Licenses">Licenses</a> |
| </li> |
| <li> <a href="mailing-lists.html" title="Mailing Lists">Mailing Lists</a> |
| </li> |
| <li> <a href="modules.html" title="Project Modules">Project Modules</a> |
| </li> |
| <li> <a href="plugin-management.html" title="Plugin Management">Plugin Management</a> |
| </li> |
| <li> <a href="plugins.html" title="Plugins">Plugins</a> |
| </li> |
| <li> <a href="scm.html" title="Source Code Management">Source Code Management</a> |
| </li> |
| <li> <a href="summary.html" title="Summary">Summary</a> |
| </li> |
| <li> <a href="team.html" title="Team">Team</a> |
| </li> |
| </ul> |
| </li> |
| |
| <li class="dropdown-submenu"> |
| <a href="project-reports.html" title="Project Reports">Project Reports</a> |
| <ul class="dropdown-menu"> |
| <li> <a href="xref/index.html" title="Source Xref">Source Xref</a> |
| </li> |
| <li> <a href="xref-test/index.html" title="Test Source Xref">Test Source Xref</a> |
| </li> |
| <li> <a href="checkstyle-aggregate.html" title="Checkstyle">Checkstyle</a> |
| </li> |
| <li> <a href="cpd.html" title="CPD">CPD</a> |
| </li> |
| <li> <a href="pmd.html" title="PMD">PMD</a> |
| </li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| </ul> |
| |
| <form id="search-form" action="https://www.google.com/search" method="get" class="navbar-search pull-right" > |
| |
| <input value="omid.incubator.apache.org" name="sitesearch" type="hidden"/> |
| <input class="search-query" name="q" id="query" type="text" /> |
| </form> |
| <script type="text/javascript" src="https://cse.google.com/brand?form=search-form"></script> |
| |
| |
| |
| |
| <ul class="nav pull-right"><li> |
| |
| <a href="https://twitter.com/apacheomid" class="twitter-follow-button" data-show-count="true" data-align="right" data-size="large" data-show-screen-name="true" data-lang="en">Follow apacheomid</a> |
| <script type="text/javascript">!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0];if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src="//platform.twitter.com/widgets.js";fjs.parentNode.insertBefore(js,fjs);}}(document,"script","twitter-wjs");</script> |
| |
| </li></ul> |
| |
| |
| </div> |
| |
| </div> |
| </div> |
| </div> |
| |
| <div class="container"> |
| <div id="banner"> |
| <div class="pull-left"> |
| <a href="./" id="bannerLeft"> |
| <img src="images/omid-logo.png" alt="Omid" width="200"/> |
| </a> |
| </div> |
| <div class="pull-right"> </div> |
| <div class="clear"><hr/></div> |
| </div> |
| |
| <div id="breadcrumbs"> |
| <ul class="breadcrumb"> |
| |
| |
| |
| <li id="publishDate" class="pull-right">Last Published: 2020-11-23</li> |
| |
| </ul> |
| </div> |
| |
| |
| |
| <div id="bodyColumn" > |
| |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1>Management of Client Failures</h1> |
| <p>Upon user application crashes, Transactional Clients may leave some orphaned data in the datastore, even though the data in the datastore is kept in a consistent state. When another Transactional Client comes across this data, it will check whether the associated transaction was committed or not. If the data belongs to a transaction which has not been committed, and the transaction id is lower than the low watermark, then the data is deleted, since it belongs to a transaction that can never be committed.</p> |
| <p>If data belongs to a transaction that was already committed, this means that the Transactional Client crashed between sending the commit request and completing the transaction in the commit table. This means that shadow cells have not been deleted. The Transactional Client reading will write the shadow cell for the orphan data. However, it is not able to complete the transaction in the commit table as it can not guarantee that shadow cells have been written for all cells in the transaction, since only the crashed client knew which cells it had written to. Consequently, this means that the commit table can grow indefinitely. Initially, this flaw should not cause many problems, since:</p> |
| <ul> |
| |
| <li>the amount of data stored is small (16 bytes)</li> |
| <li>the number of clients expected to fail between commit and completion is low * the data is stored on persistent storage. It is not bounded by memory.</li> |
| </ul> |
| <p>For example, if 1% of all Transactional Clients are expected to crash at this exact point in time, while the system is writing, on average, 100,000 transactions per second, the amount of data required to store these commit table entries for a whole year would be around 500 GB. A single disk could hold this.</p> |
| <p>In any case, it would be desirable a mechanism to sanitize the Commit Table. If we can guarantee that all orphan data has been cleaned-up up to a certain low watermark, then all commit table entries whose start timestamp is lower than this low watermark can be deleted.</p> |
| <p>In the case of HBase, there is already have a compactor which proactively cleans up transactions older than the low watermark. This could easily be extended to store the low watermark for each region it compacts, so then another process could clean up the commit table based on the minimum low watermark that was stored.</p> |
| </div> |
| </div> |
| |
| <hr/> |
| |
| <footer> |
| <div class="container"> |
| <div class="row"> |
| <p >Copyright © 2011–2020 |
| <a href="http://www.apache.org">Apache Software Foundation</a>. |
| All rights reserved. |
| </p> |
| </div> |
| |
| <p id="poweredBy" class="pull-right"> |
| <a href="http://maven.apache.org/" title="Maven" class="builtBy"> |
| <img class="builtBy" alt="Maven" src="http://maven.apache.org/images/logos/maven-feather.png" /> |
| </a> |
| </p> |
| |
| </div> |
| </footer> |
| </body> |
| </html> |
