| <!DOCTYPE html> |
| <html> |
| |
| |
| |
| |
| <head> |
| <meta charset="utf-8"> |
| <meta http-equiv="X-UA-Compatible" content="IE=edge"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| <meta name="description" content="The Apache Cassandra database is the right choice when you need scalability and high availability without compromising performance. Linear scalability and proven fault-tolerance on commodity hardware or cloud infrastructure make it the perfect platform for mission-critical data. Cassandra's support for replicating across multiple datacenters is best-in-class, providing lower latency for your users and the peace of mind of knowing that you can survive regional outages. |
| "> |
| <meta name="keywords" content="cassandra, apache, apache cassandra, distributed storage, key value store, scalability, bigtable, dynamo" /> |
| <meta name="robots" content="index,follow" /> |
| <meta name="language" content="en" /> |
| |
| <title>Documentation</title> |
| |
| <link rel="canonical" href="http://cassandra.apache.org/doc/4.0-alpha3/faq/"> |
| |
| <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap.min.css" integrity="sha384-1q8mTJOASx8j1Au+a5WDVnPi2lkFfwwEAa8hDDdjZlpLegxhjVME1fgjWPGmkzs7" crossorigin="anonymous"> |
| <link rel="stylesheet" href="./../../../css/style.css"> |
| |
| <link rel="stylesheet" href="./../../../css/sphinx.css"> |
| |
| |
| <link rel="top" title="Apache Cassandra Documentation v4.0-alpha3" href="../index.html"/> <link rel="next" title="Third-Party Plugins" href="../plugins/index.html"/> <link rel="prev" title="Release Process" href="../development/release_process.html"/> |
| <link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.2.0/css/all.css" integrity="sha384-hWVjflwFxL6sNzntih27bfxkr27PmbbK/iSvJ+a4+0owXq79v+lsFkW54bOGbiDQ" crossorigin="anonymous"> |
| |
| <link type="application/atom+xml" rel="alternate" href="http://cassandra.apache.org/feed.xml" title="Apache Cassandra Website" /> |
| </head> |
| |
| <body> |
| <!-- breadcrumbs --> |
| <div class="topnav"> |
| <div class="container breadcrumb-container"> |
| <ul class="breadcrumb"> |
| <li> |
| <div class="dropdown"> |
| <img class="asf-logo" src="./../../../img/asf_feather.png" /> |
| <a data-toggle="dropdown" href="#">Apache Software Foundation <span class="caret"></span></a> |
| <ul class="dropdown-menu" role="menu" aria-labelledby="dLabel"> |
| <li><a href="http://www.apache.org">Apache Homepage</a></li> |
| <li><a href="http://www.apache.org/licenses/">License</a></li> |
| <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li> |
| <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li> |
| <li><a href="http://www.apache.org/security/">Security</a></li> |
| </ul> |
| </div> |
| </li> |
| |
| |
| <li><a href="./../../../">Apache Cassandra</a></li> |
| |
| |
| |
| |
| <li><a href="./../../../doc">Documentation</a></li> |
| |
| |
| |
| |
| |
| |
| <li>Frequently Asked Questions</li> |
| |
| </ul> |
| </div> |
| |
| <!-- navbar --> |
| <nav class="navbar navbar-default navbar-static-top" role="navigation"> |
| <div class="container"> |
| <div class="navbar-header"> |
| <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#cassandra-menu" aria-expanded="false"> |
| <span class="sr-only">Toggle navigation</span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| </button> |
| <a class="navbar-brand" href="./../../../"><img src="./../../../img/cassandra_logo.png" alt="Apache Cassandra logo" /></a> |
| </div><!-- /.navbar-header --> |
| |
| <div id="cassandra-menu" class="collapse navbar-collapse"> |
| <ul class="nav navbar-nav navbar-right"> |
| <li><a href="./../../../">Home</a></li> |
| <li><a href="./../../../download/">Download</a></li> |
| <li><a href="./../../../doc/">Documentation</a></li> |
| <li><a href="./../../../community/">Community</a></li> |
| <li> |
| <a href="./../../../blog/">Blog</a> |
| </li> |
| </ul> |
| </div><!-- /#cassandra-menu --> |
| |
| |
| </div> |
| </nav><!-- /.navbar --> |
| </div><!-- /.topnav --> |
| |
| <div class="container-fluid"> |
| <div class="row"> |
| <div class="col-md-3"> |
| <div class="doc-navigation"> |
| <div class="doc-menu" role="navigation"> |
| <div class="navbar-header"> |
| <button type="button" class="pull-left navbar-toggle" data-toggle="collapse" data-target=".sidebar-navbar-collapse"> |
| <span class="sr-only">Toggle navigation</span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| </button> |
| </div> |
| <div class="navbar-collapse collapse sidebar-navbar-collapse"> |
| <form id="doc-search-form" class="navbar-form" action="../search.html" method="get" role="search"> |
| <div class="form-group"> |
| <input type="text" size="30" class="form-control input-sm" name="q" placeholder="Search docs"> |
| <input type="hidden" name="check_keywords" value="yes" /> |
| <input type="hidden" name="area" value="default" /> |
| </div> |
| </form> |
| |
| |
| |
| <ul class="current"> |
| <li class="toctree-l1"><a class="reference internal" href="../getting_started/index.html">Getting Started</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../architecture/index.html">Architecture</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../data_modeling/index.html">Data Modeling</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../cql/index.html">The Cassandra Query Language (CQL)</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../configuration/index.html">Configuring Cassandra</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../operating/index.html">Operating Cassandra</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../tools/index.html">Cassandra Tools</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../troubleshooting/index.html">Troubleshooting</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../development/index.html">Contributing to Cassandra</a></li> |
| <li class="toctree-l1 current"><a class="current reference internal" href="#">Frequently Asked Questions</a><ul> |
| <li class="toctree-l2"><a class="reference internal" href="#why-can-t-i-set-listen-address-to-listen-on-0-0-0-0-all-my-addresses">Why can’t I set <code class="docutils literal notranslate"><span class="pre">listen_address</span></code> to listen on 0.0.0.0 (all my addresses)?</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#what-ports-does-cassandra-use">What ports does Cassandra use?</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#what-happens-to-existing-data-in-my-cluster-when-i-add-new-nodes">What happens to existing data in my cluster when I add new nodes?</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#i-delete-data-from-cassandra-but-disk-usage-stays-the-same-what-gives">I delete data from Cassandra, but disk usage stays the same. What gives?</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#why-does-nodetool-ring-only-show-one-entry-even-though-my-nodes-logged-that-they-see-each-other-joining-the-ring">Why does nodetool ring only show one entry, even though my nodes logged that they see each other joining the ring?</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#can-i-change-the-replication-factor-a-a-keyspace-on-a-live-cluster">Can I change the replication factor (a a keyspace) on a live cluster?</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#can-i-store-large-blobs-in-cassandra">Can I Store (large) BLOBs in Cassandra?</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#nodetool-says-connection-refused-to-host-127-0-1-1-for-any-remote-host-what-gives">Nodetool says “Connection refused to host: 127.0.1.1” for any remote host. What gives?</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#will-batching-my-operations-speed-up-my-bulk-load">Will batching my operations speed up my bulk load?</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#on-rhel-nodes-are-unable-to-join-the-ring">On RHEL nodes are unable to join the ring</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#how-do-i-unsubscribe-from-the-email-list">How do I unsubscribe from the email list?</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#why-does-top-report-that-cassandra-is-using-a-lot-more-memory-than-the-java-heap-max">Why does top report that Cassandra is using a lot more memory than the Java heap max?</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#what-are-seeds">What are seeds?</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#does-single-seed-mean-single-point-of-failure">Does single seed mean single point of failure?</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#why-can-t-i-call-jmx-method-x-on-jconsole">Why can’t I call jmx method X on jconsole?</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#why-do-i-see-messages-dropped-in-the-logs">Why do I see “… messages dropped …” in the logs?</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#cassandra-dies-with-java-lang-outofmemoryerror-map-failed">Cassandra dies with <code class="docutils literal notranslate"><span class="pre">java.lang.OutOfMemoryError:</span> <span class="pre">Map</span> <span class="pre">failed</span></code></a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#what-happens-if-two-updates-are-made-with-the-same-timestamp">What happens if two updates are made with the same timestamp?</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#why-bootstrapping-a-new-node-fails-with-a-stream-failed-error">Why bootstrapping a new node fails with a “Stream failed” error?</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l1"><a class="reference internal" href="../plugins/index.html">Third-Party Plugins</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../bugs.html">Reporting Bugs</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../contactus.html">Contact us</a></li> |
| </ul> |
| |
| |
| |
| </div><!--/.nav-collapse --> |
| </div> |
| </div> |
| </div> |
| <div class="col-md-8"> |
| <div class="content doc-content"> |
| <div class="content-container"> |
| |
| <div class="section" id="frequently-asked-questions"> |
| <h1>Frequently Asked Questions<a class="headerlink" href="#frequently-asked-questions" title="Permalink to this headline">¶</a></h1> |
| <ul class="simple"> |
| <li><a class="reference internal" href="#why-cant-list-all"><span class="std std-ref">Why can’t I set listen_address to listen on 0.0.0.0 (all my addresses)?</span></a></li> |
| <li><a class="reference internal" href="#what-ports"><span class="std std-ref">What ports does Cassandra use?</span></a></li> |
| <li><a class="reference internal" href="#what-happens-on-joins"><span class="std std-ref">What happens to existing data in my cluster when I add new nodes?</span></a></li> |
| <li><a class="reference internal" href="#asynch-deletes"><span class="std std-ref">I delete data from Cassandra, but disk usage stays the same. What gives?</span></a></li> |
| <li><a class="reference internal" href="#one-entry-ring"><span class="std std-ref">Why does nodetool ring only show one entry, even though my nodes logged that they see each other joining the ring?</span></a></li> |
| <li><a class="reference internal" href="#can-large-blob"><span class="std std-ref">Can I Store (large) BLOBs in Cassandra?</span></a></li> |
| <li><a class="reference internal" href="#nodetool-connection-refused"><span class="std std-ref">Nodetool says “Connection refused to host: 127.0.1.1” for any remote host. What gives?</span></a></li> |
| <li><a class="reference internal" href="#to-batch-or-not-to-batch"><span class="std std-ref">Will batching my operations speed up my bulk load?</span></a></li> |
| <li><a class="reference internal" href="#selinux"><span class="std std-ref">On RHEL nodes are unable to join the ring</span></a></li> |
| <li><a class="reference internal" href="#how-to-unsubscribe"><span class="std std-ref">How do I unsubscribe from the email list?</span></a></li> |
| <li><a class="reference internal" href="#cassandra-eats-all-my-memory"><span class="std std-ref">Why does top report that Cassandra is using a lot more memory than the Java heap max?</span></a></li> |
| <li><a class="reference internal" href="#what-are-seeds"><span class="std std-ref">What are seeds?</span></a></li> |
| <li><a class="reference internal" href="#are-seeds-spof"><span class="std std-ref">Does single seed mean single point of failure?</span></a></li> |
| <li><a class="reference internal" href="#why-message-dropped"><span class="std std-ref">Why do I see “… messages dropped …” in the logs?</span></a></li> |
| <li><a class="reference internal" href="#oom-map-failed"><span class="std std-ref">Cassandra dies with java.lang.OutOfMemoryError: Map failed</span></a></li> |
| <li><a class="reference internal" href="#what-on-same-timestamp-update"><span class="std std-ref">What happens if two updates are made with the same timestamp?</span></a></li> |
| <li><a class="reference internal" href="#why-bootstrapping-stream-error"><span class="std std-ref">Why bootstrapping a new node fails with a “Stream failed” error?</span></a></li> |
| </ul> |
| <div class="section" id="why-can-t-i-set-listen-address-to-listen-on-0-0-0-0-all-my-addresses"> |
| <span id="why-cant-list-all"></span><h2>Why can’t I set <code class="docutils literal notranslate"><span class="pre">listen_address</span></code> to listen on 0.0.0.0 (all my addresses)?<a class="headerlink" href="#why-can-t-i-set-listen-address-to-listen-on-0-0-0-0-all-my-addresses" title="Permalink to this headline">¶</a></h2> |
| <p>Cassandra is a gossip-based distributed system and <code class="docutils literal notranslate"><span class="pre">listen_address</span></code> is the address a node tells other nodes to reach |
| it at. Telling other nodes “contact me on any of my addresses” is a bad idea; if different nodes in the cluster pick |
| different addresses for you, Bad Things happen.</p> |
| <p>If you don’t want to manually specify an IP to <code class="docutils literal notranslate"><span class="pre">listen_address</span></code> for each node in your cluster (understandable!), leave |
| it blank and Cassandra will use <code class="docutils literal notranslate"><span class="pre">InetAddress.getLocalHost()</span></code> to pick an address. Then it’s up to you or your ops team |
| to make things resolve correctly (<code class="docutils literal notranslate"><span class="pre">/etc/hosts/</span></code>, dns, etc).</p> |
| <p>One exception to this process is JMX, which by default binds to 0.0.0.0 (Java bug 6425769).</p> |
| <p>See <a class="reference external" href="https://issues.apache.org/jira/browse/CASSANDRA-256">CASSANDRA-256</a> and <a class="reference external" href="https://issues.apache.org/jira/browse/CASSANDRA-43">CASSANDRA-43</a> for more gory details.</p> |
| </div> |
| <div class="section" id="what-ports-does-cassandra-use"> |
| <span id="what-ports"></span><h2>What ports does Cassandra use?<a class="headerlink" href="#what-ports-does-cassandra-use" title="Permalink to this headline">¶</a></h2> |
| <p>By default, Cassandra uses 7000 for cluster communication (7001 if SSL is enabled), 9042 for native protocol clients, |
| and 7199 for JMX. The internode communication and native protocol ports |
| are configurable in the <a class="reference internal" href="../configuration/cassandra_config_file.html#cassandra-yaml"><span class="std std-ref">Cassandra Configuration File</span></a>. The JMX port is configurable in <code class="docutils literal notranslate"><span class="pre">cassandra-env.sh</span></code> (through JVM |
| options). All ports are TCP.</p> |
| </div> |
| <div class="section" id="what-happens-to-existing-data-in-my-cluster-when-i-add-new-nodes"> |
| <span id="what-happens-on-joins"></span><h2>What happens to existing data in my cluster when I add new nodes?<a class="headerlink" href="#what-happens-to-existing-data-in-my-cluster-when-i-add-new-nodes" title="Permalink to this headline">¶</a></h2> |
| <p>When a new nodes joins a cluster, it will automatically contact the other nodes in the cluster and copy the right data |
| to itself. See <a class="reference internal" href="../operating/topo_changes.html#topology-changes"><span class="std std-ref">Adding, replacing, moving and removing nodes</span></a>.</p> |
| </div> |
| <div class="section" id="i-delete-data-from-cassandra-but-disk-usage-stays-the-same-what-gives"> |
| <span id="asynch-deletes"></span><h2>I delete data from Cassandra, but disk usage stays the same. What gives?<a class="headerlink" href="#i-delete-data-from-cassandra-but-disk-usage-stays-the-same-what-gives" title="Permalink to this headline">¶</a></h2> |
| <p>Data you write to Cassandra gets persisted to SSTables. Since SSTables are immutable, the data can’t actually be removed |
| when you perform a delete, instead, a marker (also called a “tombstone”) is written to indicate the value’s new status. |
| Never fear though, on the first compaction that occurs between the data and the tombstone, the data will be expunged |
| completely and the corresponding disk space recovered. See <a class="reference internal" href="../operating/compaction.html#compaction"><span class="std std-ref">Compaction</span></a> for more detail.</p> |
| </div> |
| <div class="section" id="why-does-nodetool-ring-only-show-one-entry-even-though-my-nodes-logged-that-they-see-each-other-joining-the-ring"> |
| <span id="one-entry-ring"></span><h2>Why does nodetool ring only show one entry, even though my nodes logged that they see each other joining the ring?<a class="headerlink" href="#why-does-nodetool-ring-only-show-one-entry-even-though-my-nodes-logged-that-they-see-each-other-joining-the-ring" title="Permalink to this headline">¶</a></h2> |
| <p>This happens when you have the same token assigned to each node. Don’t do that.</p> |
| <p>Most often this bites people who deploy by installing Cassandra on a VM (especially when using the Debian package, which |
| auto-starts Cassandra after installation, thus generating and saving a token), then cloning that VM to other nodes.</p> |
| <p>The easiest fix is to wipe the data and commitlog directories, thus making sure that each node will generate a random |
| token on the next restart.</p> |
| </div> |
| <div class="section" id="can-i-change-the-replication-factor-a-a-keyspace-on-a-live-cluster"> |
| <span id="change-replication-factor"></span><h2>Can I change the replication factor (a a keyspace) on a live cluster?<a class="headerlink" href="#can-i-change-the-replication-factor-a-a-keyspace-on-a-live-cluster" title="Permalink to this headline">¶</a></h2> |
| <p>Yes, but it will require running a full repair (or cleanup) to change the replica count of existing data:</p> |
| <ul class="simple"> |
| <li><a class="reference internal" href="../cql/ddl.html#alter-keyspace-statement"><span class="std std-ref">Alter</span></a> the replication factor for desired keyspace (using cqlsh for instance).</li> |
| <li>If you’re reducing the replication factor, run <code class="docutils literal notranslate"><span class="pre">nodetool</span> <span class="pre">cleanup</span></code> on the cluster to remove surplus replicated data. |
| Cleanup runs on a per-node basis.</li> |
| <li>If you’re increasing the replication factor, run <code class="docutils literal notranslate"><span class="pre">nodetool</span> <span class="pre">repair</span> <span class="pre">-full</span></code> to ensure data is replicated according to the new |
| configuration. Repair runs on a per-replica set basis. This is an intensive process that may result in adverse cluster |
| performance. It’s highly recommended to do rolling repairs, as an attempt to repair the entire cluster at once will |
| most likely swamp it. Note that you will need to run a full repair (<code class="docutils literal notranslate"><span class="pre">-full</span></code>) to make sure that already repaired |
| sstables are not skipped.</li> |
| </ul> |
| </div> |
| <div class="section" id="can-i-store-large-blobs-in-cassandra"> |
| <span id="can-large-blob"></span><h2>Can I Store (large) BLOBs in Cassandra?<a class="headerlink" href="#can-i-store-large-blobs-in-cassandra" title="Permalink to this headline">¶</a></h2> |
| <p>Cassandra isn’t optimized for large file or BLOB storage and a single <code class="docutils literal notranslate"><span class="pre">blob</span></code> value is always read and send to the |
| client entirely. As such, storing small blobs (less than single digit MB) should not be a problem, but it is advised to |
| manually split large blobs into smaller chunks.</p> |
| <p>Please note in particular that by default, any value greater than 16MB will be rejected by Cassandra due the |
| <code class="docutils literal notranslate"><span class="pre">max_mutation_size_in_kb</span></code> configuration of the <a class="reference internal" href="../configuration/cassandra_config_file.html#cassandra-yaml"><span class="std std-ref">Cassandra Configuration File</span></a> file (which default to half of |
| <code class="docutils literal notranslate"><span class="pre">commitlog_segment_size_in_mb</span></code>, which itself default to 32MB).</p> |
| </div> |
| <div class="section" id="nodetool-says-connection-refused-to-host-127-0-1-1-for-any-remote-host-what-gives"> |
| <span id="nodetool-connection-refused"></span><h2>Nodetool says “Connection refused to host: 127.0.1.1” for any remote host. What gives?<a class="headerlink" href="#nodetool-says-connection-refused-to-host-127-0-1-1-for-any-remote-host-what-gives" title="Permalink to this headline">¶</a></h2> |
| <p>Nodetool relies on JMX, which in turn relies on RMI, which in turn sets up its own listeners and connectors as needed on |
| each end of the exchange. Normally all of this happens behind the scenes transparently, but incorrect name resolution |
| for either the host connecting, or the one being connected to, can result in crossed wires and confusing exceptions.</p> |
| <p>If you are not using DNS, then make sure that your <code class="docutils literal notranslate"><span class="pre">/etc/hosts</span></code> files are accurate on both ends. If that fails, try |
| setting the <code class="docutils literal notranslate"><span class="pre">-Djava.rmi.server.hostname=<public</span> <span class="pre">name></span></code> JVM option near the bottom of <code class="docutils literal notranslate"><span class="pre">cassandra-env.sh</span></code> to an |
| interface that you can reach from the remote machine.</p> |
| </div> |
| <div class="section" id="will-batching-my-operations-speed-up-my-bulk-load"> |
| <span id="to-batch-or-not-to-batch"></span><h2>Will batching my operations speed up my bulk load?<a class="headerlink" href="#will-batching-my-operations-speed-up-my-bulk-load" title="Permalink to this headline">¶</a></h2> |
| <p>No. Using batches to load data will generally just add “spikes” of latency. Use asynchronous INSERTs instead, or use |
| true <a class="reference internal" href="../operating/bulk_loading.html#bulk-loading"><span class="std std-ref">Bulk Loading</span></a>.</p> |
| <p>An exception is batching updates to a single partition, which can be a Good Thing (as long as the size of a single batch |
| stay reasonable). But never ever blindly batch everything!</p> |
| </div> |
| <div class="section" id="on-rhel-nodes-are-unable-to-join-the-ring"> |
| <span id="selinux"></span><h2>On RHEL nodes are unable to join the ring<a class="headerlink" href="#on-rhel-nodes-are-unable-to-join-the-ring" title="Permalink to this headline">¶</a></h2> |
| <p>Check if <a class="reference external" href="https://en.wikipedia.org/wiki/Security-Enhanced_Linux">SELinux</a> is on; if it is, turn it off.</p> |
| </div> |
| <div class="section" id="how-do-i-unsubscribe-from-the-email-list"> |
| <span id="how-to-unsubscribe"></span><h2>How do I unsubscribe from the email list?<a class="headerlink" href="#how-do-i-unsubscribe-from-the-email-list" title="Permalink to this headline">¶</a></h2> |
| <p>Send an email to <code class="docutils literal notranslate"><span class="pre">user-unsubscribe@cassandra.apache.org</span></code>.</p> |
| </div> |
| <div class="section" id="why-does-top-report-that-cassandra-is-using-a-lot-more-memory-than-the-java-heap-max"> |
| <span id="cassandra-eats-all-my-memory"></span><h2>Why does top report that Cassandra is using a lot more memory than the Java heap max?<a class="headerlink" href="#why-does-top-report-that-cassandra-is-using-a-lot-more-memory-than-the-java-heap-max" title="Permalink to this headline">¶</a></h2> |
| <p>Cassandra uses <a class="reference external" href="https://en.wikipedia.org/wiki/Memory-mapped_file">Memory Mapped Files</a> (mmap) internally. That is, we |
| use the operating system’s virtual memory system to map a number of on-disk files into the Cassandra process’ address |
| space. This will “use” virtual memory; i.e. address space, and will be reported by tools like top accordingly, but on 64 |
| bit systems virtual address space is effectively unlimited so you should not worry about that.</p> |
| <p>What matters from the perspective of “memory use” in the sense as it is normally meant, is the amount of data allocated |
| on brk() or mmap’d /dev/zero, which represent real memory used. The key issue is that for a mmap’d file, there is never |
| a need to retain the data resident in physical memory. Thus, whatever you do keep resident in physical memory is |
| essentially just there as a cache, in the same way as normal I/O will cause the kernel page cache to retain data that |
| you read/write.</p> |
| <p>The difference between normal I/O and mmap() is that in the mmap() case the memory is actually mapped to the process, |
| thus affecting the virtual size as reported by top. The main argument for using mmap() instead of standard I/O is the |
| fact that reading entails just touching memory - in the case of the memory being resident, you just read it - you don’t |
| even take a page fault (so no overhead in entering the kernel and doing a semi-context switch). This is covered in more |
| detail <a class="reference external" href="http://www.varnish-cache.org/trac/wiki/ArchitectNotes">here</a>.</p> |
| </div> |
| <div class="section" id="what-are-seeds"> |
| <span id="id1"></span><h2>What are seeds?<a class="headerlink" href="#what-are-seeds" title="Permalink to this headline">¶</a></h2> |
| <p>Seeds are used during startup to discover the cluster.</p> |
| <p>If you configure your nodes to refer some node as seed, nodes in your ring tend to send Gossip message to seeds more |
| often (also see the <a class="reference internal" href="../architecture/dynamo.html#gossip"><span class="std std-ref">section on gossip</span></a>) than to non-seeds. In other words, seeds are worked as hubs of |
| Gossip network. With seeds, each node can detect status changes of other nodes quickly.</p> |
| <p>Seeds are also referred by new nodes on bootstrap to learn other nodes in ring. When you add a new node to ring, you |
| need to specify at least one live seed to contact. Once a node join the ring, it learns about the other nodes, so it |
| doesn’t need seed on subsequent boot.</p> |
| <p>You can make a seed a node at any time. There is nothing special about seed nodes. If you list the node in seed list it |
| is a seed</p> |
| <p>Seeds do not auto bootstrap (i.e. if a node has itself in its seed list it will not automatically transfer data to itself) |
| If you want a node to do that, bootstrap it first and then add it to seeds later. If you have no data (new install) you |
| do not have to worry about bootstrap at all.</p> |
| <p>Recommended usage of seeds:</p> |
| <ul class="simple"> |
| <li>pick two (or more) nodes per data center as seed nodes.</li> |
| <li>sync the seed list to all your nodes</li> |
| </ul> |
| </div> |
| <div class="section" id="does-single-seed-mean-single-point-of-failure"> |
| <span id="are-seeds-spof"></span><h2>Does single seed mean single point of failure?<a class="headerlink" href="#does-single-seed-mean-single-point-of-failure" title="Permalink to this headline">¶</a></h2> |
| <p>The ring can operate or boot without a seed; however, you will not be able to add new nodes to the cluster. It is |
| recommended to configure multiple seeds in production system.</p> |
| </div> |
| <div class="section" id="why-can-t-i-call-jmx-method-x-on-jconsole"> |
| <span id="cant-call-jmx-method"></span><h2>Why can’t I call jmx method X on jconsole?<a class="headerlink" href="#why-can-t-i-call-jmx-method-x-on-jconsole" title="Permalink to this headline">¶</a></h2> |
| <p>Some of JMX operations use array argument and as jconsole doesn’t support array argument, those operations can’t be |
| called with jconsole (the buttons are inactive for them). You need to write a JMX client to call such operations or need |
| array-capable JMX monitoring tool.</p> |
| </div> |
| <div class="section" id="why-do-i-see-messages-dropped-in-the-logs"> |
| <span id="why-message-dropped"></span><h2>Why do I see “… messages dropped …” in the logs?<a class="headerlink" href="#why-do-i-see-messages-dropped-in-the-logs" title="Permalink to this headline">¶</a></h2> |
| <p>This is a symptom of load shedding – Cassandra defending itself against more requests than it can handle.</p> |
| <p>Internode messages which are received by a node, but do not get not to be processed within their proper timeout (see |
| <code class="docutils literal notranslate"><span class="pre">read_request_timeout</span></code>, <code class="docutils literal notranslate"><span class="pre">write_request_timeout</span></code>, … in the <a class="reference internal" href="../configuration/cassandra_config_file.html#cassandra-yaml"><span class="std std-ref">Cassandra Configuration File</span></a>), are dropped rather than |
| processed (since the as the coordinator node will no longer be waiting for a response).</p> |
| <p>For writes, this means that the mutation was not applied to all replicas it was sent to. The inconsistency will be |
| repaired by read repair, hints or a manual repair. The write operation may also have timeouted as a result.</p> |
| <p>For reads, this means a read request may not have completed.</p> |
| <p>Load shedding is part of the Cassandra architecture, if this is a persistent issue it is generally a sign of an |
| overloaded node or cluster.</p> |
| </div> |
| <div class="section" id="cassandra-dies-with-java-lang-outofmemoryerror-map-failed"> |
| <span id="oom-map-failed"></span><h2>Cassandra dies with <code class="docutils literal notranslate"><span class="pre">java.lang.OutOfMemoryError:</span> <span class="pre">Map</span> <span class="pre">failed</span></code><a class="headerlink" href="#cassandra-dies-with-java-lang-outofmemoryerror-map-failed" title="Permalink to this headline">¶</a></h2> |
| <p>If Cassandra is dying <strong>specifically</strong> with the “Map failed” message, it means the OS is denying java the ability to |
| lock more memory. In linux, this typically means memlock is limited. Check <code class="docutils literal notranslate"><span class="pre">/proc/<pid</span> <span class="pre">of</span> <span class="pre">cassandra>/limits</span></code> to verify |
| this and raise it (eg, via ulimit in bash). You may also need to increase <code class="docutils literal notranslate"><span class="pre">vm.max_map_count.</span></code> Note that the debian |
| package handles this for you automatically.</p> |
| </div> |
| <div class="section" id="what-happens-if-two-updates-are-made-with-the-same-timestamp"> |
| <span id="what-on-same-timestamp-update"></span><h2>What happens if two updates are made with the same timestamp?<a class="headerlink" href="#what-happens-if-two-updates-are-made-with-the-same-timestamp" title="Permalink to this headline">¶</a></h2> |
| <p>Updates must be commutative, since they may arrive in different orders on different replicas. As long as Cassandra has a |
| deterministic way to pick the winner (in a timestamp tie), the one selected is as valid as any other, and the specifics |
| should be treated as an implementation detail. That said, in the case of a timestamp tie, Cassandra follows two rules: |
| first, deletes take precedence over inserts/updates. Second, if there are two updates, the one with the lexically larger |
| value is selected.</p> |
| </div> |
| <div class="section" id="why-bootstrapping-a-new-node-fails-with-a-stream-failed-error"> |
| <span id="why-bootstrapping-stream-error"></span><h2>Why bootstrapping a new node fails with a “Stream failed” error?<a class="headerlink" href="#why-bootstrapping-a-new-node-fails-with-a-stream-failed-error" title="Permalink to this headline">¶</a></h2> |
| <p>Two main possibilities:</p> |
| <ol class="arabic simple"> |
| <li>the GC may be creating long pauses disrupting the streaming process</li> |
| <li>compactions happening in the background hold streaming long enough that the TCP connection fails</li> |
| </ol> |
| <p>In the first case, regular GC tuning advices apply. In the second case, you need to set TCP keepalive to a lower value |
| (default is very high on Linux). Try to just run the following:</p> |
| <div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ sudo /sbin/sysctl -w net.ipv4.tcp_keepalive_time=60 net.ipv4.tcp_keepalive_intvl=60 net.ipv4.tcp_keepalive_probes=5 |
| </pre></div> |
| </div> |
| <p>To make those settings permanent, add them to your <code class="docutils literal notranslate"><span class="pre">/etc/sysctl.conf</span></code> file.</p> |
| <p>Note: <a class="reference external" href="https://cloud.google.com/compute/">GCE</a>’s firewall will always interrupt TCP connections that are inactive for |
| more than 10 min. Running the above command is highly recommended in that environment.</p> |
| </div> |
| </div> |
| |
| |
| |
| |
| <div class="doc-prev-next-links" role="navigation" aria-label="footer navigation"> |
| |
| <a href="../plugins/index.html" class="btn btn-default pull-right " role="button" title="Third-Party Plugins" accesskey="n">Next <span class="glyphicon glyphicon-circle-arrow-right" aria-hidden="true"></span></a> |
| |
| |
| <a href="../development/release_process.html" class="btn btn-default" role="button" title="Release Process" accesskey="p"><span class="glyphicon glyphicon-circle-arrow-left" aria-hidden="true"></span> Previous</a> |
| |
| </div> |
| |
| </div> |
| </div> |
| </div> |
| </div> |
| </div> |
| |
| <hr /> |
| |
| <footer> |
| <div class="container"> |
| <div class="col-md-4 social-blk"> |
| <span class="social"> |
| <a href="https://twitter.com/cassandra" |
| class="twitter-follow-button" |
| data-show-count="false" data-size="large">Follow @cassandra</a> |
| <script>!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0],p=/^http:/.test(d.location)?'http':'https';if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src=p+'://platform.twitter.com/widgets.js';fjs.parentNode.insertBefore(js,fjs);}}(document, 'script', 'twitter-wjs');</script> |
| <a href="https://twitter.com/intent/tweet?button_hashtag=cassandra" |
| class="twitter-hashtag-button" |
| data-size="large" |
| data-related="ApacheCassandra">Tweet #cassandra</a> |
| <script>!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0],p=/^http:/.test(d.location)?'http':'https';if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src=p+'://platform.twitter.com/widgets.js';fjs.parentNode.insertBefore(js,fjs);}}(document, 'script', 'twitter-wjs');</script> |
| |
| </span> |
| <a class="subscribe-rss icon-link" href="/feed.xml" title="Subscribe to Blog via RSS"> |
| <span><i class="fa fa-rss"></i></span> |
| </a> |
| </div> |
| |
| <div class="col-md-8 trademark"> |
| <p>© 2016 <a href="http://apache.org">The Apache Software Foundation</a>. |
| Apache, the Apache feather logo, and Apache Cassandra are trademarks of The Apache Software Foundation. |
| <p> |
| </div> |
| </div><!-- /.container --> |
| </footer> |
| |
| <!-- Javascript. Placed here so pages load faster --> |
| <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script> |
| <script src="./../../../js/underscore-min.js"></script> |
| <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/js/bootstrap.min.js" integrity="sha384-0mSbJDEHialfmuBBQP6A4Qrprq5OVfW37PRR3j5ELqxss1yVqOtnepnHVP9aJ7xS" crossorigin="anonymous"></script> |
| |
| |
| <script src="./../../../js/doctools.js"></script> |
| <script src="./../../../js/searchtools.js"></script> |
| |
| <script type="text/javascript"> var DOCUMENTATION_OPTIONS = { URL_ROOT: "", VERSION: "", COLLAPSE_INDEX: false, FILE_SUFFIX: ".html", HAS_SOURCE: false, SOURCELINK_SUFFIX: ".txt" }; </script> |
| |
| |
| |
| <script type="text/javascript"> |
| var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www."); |
| document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E")); |
| |
| try { |
| var pageTracker = _gat._getTracker("UA-11583863-1"); |
| pageTracker._trackPageview(); |
| } catch(err) {} |
| </script> |
| |
| |
| </body> |
| </html> |