| --- |
| layout: docpage |
| |
| title: "Documentation" |
| |
| is_homepage: false |
| is_sphinx_doc: true |
| |
| doc-title: "Frequently Asked Questions" |
| doc-header-links: ' |
| <link rel="top" title="Apache Cassandra Documentation v4.0-alpha1" href="../index.html"/> |
| <link rel="next" title="Third-Party Plugins" href="../plugins/index.html"/> |
| <link rel="prev" title="Release Process" href="../development/release_process.html"/> |
| ' |
| doc-search-path: "../search.html" |
| |
| extra-footer: ' |
| <script type="text/javascript"> |
| var DOCUMENTATION_OPTIONS = { |
| URL_ROOT: "", |
| VERSION: "", |
| COLLAPSE_INDEX: false, |
| FILE_SUFFIX: ".html", |
| HAS_SOURCE: false, |
| SOURCELINK_SUFFIX: ".txt" |
| }; |
| </script> |
| ' |
| |
| --- |
| <div class="container-fluid"> |
| <div class="row"> |
| <div class="col-md-3"> |
| <div class="doc-navigation"> |
| <div class="doc-menu" role="navigation"> |
| <div class="navbar-header"> |
| <button type="button" class="pull-left navbar-toggle" data-toggle="collapse" data-target=".sidebar-navbar-collapse"> |
| <span class="sr-only">Toggle navigation</span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| </button> |
| </div> |
| <div class="navbar-collapse collapse sidebar-navbar-collapse"> |
| <form id="doc-search-form" class="navbar-form" action="../search.html" method="get" role="search"> |
| <div class="form-group"> |
| <input type="text" size="30" class="form-control input-sm" name="q" placeholder="Search docs"> |
| <input type="hidden" name="check_keywords" value="yes" /> |
| <input type="hidden" name="area" value="default" /> |
| </div> |
| </form> |
| |
| |
| |
| <ul class="current"> |
| <li class="toctree-l1"><a class="reference internal" href="../getting_started/index.html">Getting Started</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../architecture/index.html">Architecture</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../data_modeling/index.html">Data Modeling</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../cql/index.html">The Cassandra Query Language (CQL)</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../configuration/index.html">Configuring Cassandra</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../operating/index.html">Operating Cassandra</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../tools/index.html">Cassandra Tools</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../troubleshooting/index.html">Troubleshooting</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../development/index.html">Contributing to Cassandra</a></li> |
| <li class="toctree-l1 current"><a class="current reference internal" href="#">Frequently Asked Questions</a><ul> |
| <li class="toctree-l2"><a class="reference internal" href="#why-can-t-i-set-listen-address-to-listen-on-0-0-0-0-all-my-addresses">Why can’t I set <code class="docutils literal notranslate"><span class="pre">listen_address</span></code> to listen on 0.0.0.0 (all my addresses)?</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#what-ports-does-cassandra-use">What ports does Cassandra use?</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#what-happens-to-existing-data-in-my-cluster-when-i-add-new-nodes">What happens to existing data in my cluster when I add new nodes?</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#i-delete-data-from-cassandra-but-disk-usage-stays-the-same-what-gives">I delete data from Cassandra, but disk usage stays the same. What gives?</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#why-does-nodetool-ring-only-show-one-entry-even-though-my-nodes-logged-that-they-see-each-other-joining-the-ring">Why does nodetool ring only show one entry, even though my nodes logged that they see each other joining the ring?</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#can-i-change-the-replication-factor-a-a-keyspace-on-a-live-cluster">Can I change the replication factor (a a keyspace) on a live cluster?</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#can-i-store-large-blobs-in-cassandra">Can I Store (large) BLOBs in Cassandra?</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#nodetool-says-connection-refused-to-host-127-0-1-1-for-any-remote-host-what-gives">Nodetool says “Connection refused to host: 127.0.1.1” for any remote host. What gives?</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#will-batching-my-operations-speed-up-my-bulk-load">Will batching my operations speed up my bulk load?</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#on-rhel-nodes-are-unable-to-join-the-ring">On RHEL nodes are unable to join the ring</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#how-do-i-unsubscribe-from-the-email-list">How do I unsubscribe from the email list?</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#why-does-top-report-that-cassandra-is-using-a-lot-more-memory-than-the-java-heap-max">Why does top report that Cassandra is using a lot more memory than the Java heap max?</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#what-are-seeds">What are seeds?</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#does-single-seed-mean-single-point-of-failure">Does single seed mean single point of failure?</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#why-can-t-i-call-jmx-method-x-on-jconsole">Why can’t I call jmx method X on jconsole?</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#why-do-i-see-messages-dropped-in-the-logs">Why do I see “… messages dropped …” in the logs?</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#cassandra-dies-with-java-lang-outofmemoryerror-map-failed">Cassandra dies with <code class="docutils literal notranslate"><span class="pre">java.lang.OutOfMemoryError:</span> <span class="pre">Map</span> <span class="pre">failed</span></code></a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#what-happens-if-two-updates-are-made-with-the-same-timestamp">What happens if two updates are made with the same timestamp?</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="#why-bootstrapping-a-new-node-fails-with-a-stream-failed-error">Why bootstrapping a new node fails with a “Stream failed” error?</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l1"><a class="reference internal" href="../plugins/index.html">Third-Party Plugins</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../bugs.html">Reporting Bugs</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../contactus.html">Contact us</a></li> |
| </ul> |
| |
| |
| |
| </div><!--/.nav-collapse --> |
| </div> |
| </div> |
| </div> |
| <div class="col-md-8"> |
| <div class="content doc-content"> |
| <div class="content-container"> |
| |
| <div class="section" id="frequently-asked-questions"> |
| <h1>Frequently Asked Questions<a class="headerlink" href="#frequently-asked-questions" title="Permalink to this headline">¶</a></h1> |
| <ul class="simple"> |
| <li><a class="reference internal" href="#why-cant-list-all"><span class="std std-ref">Why can’t I set listen_address to listen on 0.0.0.0 (all my addresses)?</span></a></li> |
| <li><a class="reference internal" href="#what-ports"><span class="std std-ref">What ports does Cassandra use?</span></a></li> |
| <li><a class="reference internal" href="#what-happens-on-joins"><span class="std std-ref">What happens to existing data in my cluster when I add new nodes?</span></a></li> |
| <li><a class="reference internal" href="#asynch-deletes"><span class="std std-ref">I delete data from Cassandra, but disk usage stays the same. What gives?</span></a></li> |
| <li><a class="reference internal" href="#one-entry-ring"><span class="std std-ref">Why does nodetool ring only show one entry, even though my nodes logged that they see each other joining the ring?</span></a></li> |
| <li><a class="reference internal" href="#can-large-blob"><span class="std std-ref">Can I Store (large) BLOBs in Cassandra?</span></a></li> |
| <li><a class="reference internal" href="#nodetool-connection-refused"><span class="std std-ref">Nodetool says “Connection refused to host: 127.0.1.1” for any remote host. What gives?</span></a></li> |
| <li><a class="reference internal" href="#to-batch-or-not-to-batch"><span class="std std-ref">Will batching my operations speed up my bulk load?</span></a></li> |
| <li><a class="reference internal" href="#selinux"><span class="std std-ref">On RHEL nodes are unable to join the ring</span></a></li> |
| <li><a class="reference internal" href="#how-to-unsubscribe"><span class="std std-ref">How do I unsubscribe from the email list?</span></a></li> |
| <li><a class="reference internal" href="#cassandra-eats-all-my-memory"><span class="std std-ref">Why does top report that Cassandra is using a lot more memory than the Java heap max?</span></a></li> |
| <li><a class="reference internal" href="#what-are-seeds"><span class="std std-ref">What are seeds?</span></a></li> |
| <li><a class="reference internal" href="#are-seeds-spof"><span class="std std-ref">Does single seed mean single point of failure?</span></a></li> |
| <li><a class="reference internal" href="#why-message-dropped"><span class="std std-ref">Why do I see “… messages dropped …” in the logs?</span></a></li> |
| <li><a class="reference internal" href="#oom-map-failed"><span class="std std-ref">Cassandra dies with java.lang.OutOfMemoryError: Map failed</span></a></li> |
| <li><a class="reference internal" href="#what-on-same-timestamp-update"><span class="std std-ref">What happens if two updates are made with the same timestamp?</span></a></li> |
| <li><a class="reference internal" href="#why-bootstrapping-stream-error"><span class="std std-ref">Why bootstrapping a new node fails with a “Stream failed” error?</span></a></li> |
| </ul> |
| <div class="section" id="why-can-t-i-set-listen-address-to-listen-on-0-0-0-0-all-my-addresses"> |
| <span id="why-cant-list-all"></span><h2>Why can’t I set <code class="docutils literal notranslate"><span class="pre">listen_address</span></code> to listen on 0.0.0.0 (all my addresses)?<a class="headerlink" href="#why-can-t-i-set-listen-address-to-listen-on-0-0-0-0-all-my-addresses" title="Permalink to this headline">¶</a></h2> |
| <p>Cassandra is a gossip-based distributed system and <code class="docutils literal notranslate"><span class="pre">listen_address</span></code> is the address a node tells other nodes to reach |
| it at. Telling other nodes “contact me on any of my addresses” is a bad idea; if different nodes in the cluster pick |
| different addresses for you, Bad Things happen.</p> |
| <p>If you don’t want to manually specify an IP to <code class="docutils literal notranslate"><span class="pre">listen_address</span></code> for each node in your cluster (understandable!), leave |
| it blank and Cassandra will use <code class="docutils literal notranslate"><span class="pre">InetAddress.getLocalHost()</span></code> to pick an address. Then it’s up to you or your ops team |
| to make things resolve correctly (<code class="docutils literal notranslate"><span class="pre">/etc/hosts/</span></code>, dns, etc).</p> |
| <p>One exception to this process is JMX, which by default binds to 0.0.0.0 (Java bug 6425769).</p> |
| <p>See <a class="reference external" href="https://issues.apache.org/jira/browse/CASSANDRA-256">CASSANDRA-256</a> and <a class="reference external" href="https://issues.apache.org/jira/browse/CASSANDRA-43">CASSANDRA-43</a> for more gory details.</p> |
| </div> |
| <div class="section" id="what-ports-does-cassandra-use"> |
| <span id="what-ports"></span><h2>What ports does Cassandra use?<a class="headerlink" href="#what-ports-does-cassandra-use" title="Permalink to this headline">¶</a></h2> |
| <p>By default, Cassandra uses 7000 for cluster communication (7001 if SSL is enabled), 9042 for native protocol clients, |
| and 7199 for JMX. The internode communication and native protocol ports |
| are configurable in the <a class="reference internal" href="../configuration/cassandra_config_file.html#cassandra-yaml"><span class="std std-ref">Cassandra Configuration File</span></a>. The JMX port is configurable in <code class="docutils literal notranslate"><span class="pre">cassandra-env.sh</span></code> (through JVM |
| options). All ports are TCP.</p> |
| </div> |
| <div class="section" id="what-happens-to-existing-data-in-my-cluster-when-i-add-new-nodes"> |
| <span id="what-happens-on-joins"></span><h2>What happens to existing data in my cluster when I add new nodes?<a class="headerlink" href="#what-happens-to-existing-data-in-my-cluster-when-i-add-new-nodes" title="Permalink to this headline">¶</a></h2> |
| <p>When a new nodes joins a cluster, it will automatically contact the other nodes in the cluster and copy the right data |
| to itself. See <a class="reference internal" href="../operating/topo_changes.html#topology-changes"><span class="std std-ref">Adding, replacing, moving and removing nodes</span></a>.</p> |
| </div> |
| <div class="section" id="i-delete-data-from-cassandra-but-disk-usage-stays-the-same-what-gives"> |
| <span id="asynch-deletes"></span><h2>I delete data from Cassandra, but disk usage stays the same. What gives?<a class="headerlink" href="#i-delete-data-from-cassandra-but-disk-usage-stays-the-same-what-gives" title="Permalink to this headline">¶</a></h2> |
| <p>Data you write to Cassandra gets persisted to SSTables. Since SSTables are immutable, the data can’t actually be removed |
| when you perform a delete, instead, a marker (also called a “tombstone”) is written to indicate the value’s new status. |
| Never fear though, on the first compaction that occurs between the data and the tombstone, the data will be expunged |
| completely and the corresponding disk space recovered. See <a class="reference internal" href="../operating/compaction.html#compaction"><span class="std std-ref">Compaction</span></a> for more detail.</p> |
| </div> |
| <div class="section" id="why-does-nodetool-ring-only-show-one-entry-even-though-my-nodes-logged-that-they-see-each-other-joining-the-ring"> |
| <span id="one-entry-ring"></span><h2>Why does nodetool ring only show one entry, even though my nodes logged that they see each other joining the ring?<a class="headerlink" href="#why-does-nodetool-ring-only-show-one-entry-even-though-my-nodes-logged-that-they-see-each-other-joining-the-ring" title="Permalink to this headline">¶</a></h2> |
| <p>This happens when you have the same token assigned to each node. Don’t do that.</p> |
| <p>Most often this bites people who deploy by installing Cassandra on a VM (especially when using the Debian package, which |
| auto-starts Cassandra after installation, thus generating and saving a token), then cloning that VM to other nodes.</p> |
| <p>The easiest fix is to wipe the data and commitlog directories, thus making sure that each node will generate a random |
| token on the next restart.</p> |
| </div> |
| <div class="section" id="can-i-change-the-replication-factor-a-a-keyspace-on-a-live-cluster"> |
| <span id="change-replication-factor"></span><h2>Can I change the replication factor (a a keyspace) on a live cluster?<a class="headerlink" href="#can-i-change-the-replication-factor-a-a-keyspace-on-a-live-cluster" title="Permalink to this headline">¶</a></h2> |
| <p>Yes, but it will require running a full repair (or cleanup) to change the replica count of existing data:</p> |
| <ul class="simple"> |
| <li><a class="reference internal" href="../cql/ddl.html#alter-keyspace-statement"><span class="std std-ref">Alter</span></a> the replication factor for desired keyspace (using cqlsh for instance).</li> |
| <li>If you’re reducing the replication factor, run <code class="docutils literal notranslate"><span class="pre">nodetool</span> <span class="pre">cleanup</span></code> on the cluster to remove surplus replicated data. |
| Cleanup runs on a per-node basis.</li> |
| <li>If you’re increasing the replication factor, run <code class="docutils literal notranslate"><span class="pre">nodetool</span> <span class="pre">repair</span> <span class="pre">-full</span></code> to ensure data is replicated according to the new |
| configuration. Repair runs on a per-replica set basis. This is an intensive process that may result in adverse cluster |
| performance. It’s highly recommended to do rolling repairs, as an attempt to repair the entire cluster at once will |
| most likely swamp it. Note that you will need to run a full repair (<code class="docutils literal notranslate"><span class="pre">-full</span></code>) to make sure that already repaired |
| sstables are not skipped.</li> |
| </ul> |
| </div> |
| <div class="section" id="can-i-store-large-blobs-in-cassandra"> |
| <span id="can-large-blob"></span><h2>Can I Store (large) BLOBs in Cassandra?<a class="headerlink" href="#can-i-store-large-blobs-in-cassandra" title="Permalink to this headline">¶</a></h2> |
| <p>Cassandra isn’t optimized for large file or BLOB storage and a single <code class="docutils literal notranslate"><span class="pre">blob</span></code> value is always read and send to the |
| client entirely. As such, storing small blobs (less than single digit MB) should not be a problem, but it is advised to |
| manually split large blobs into smaller chunks.</p> |
| <p>Please note in particular that by default, any value greater than 16MB will be rejected by Cassandra due the |
| <code class="docutils literal notranslate"><span class="pre">max_mutation_size_in_kb</span></code> configuration of the <a class="reference internal" href="../configuration/cassandra_config_file.html#cassandra-yaml"><span class="std std-ref">Cassandra Configuration File</span></a> file (which default to half of |
| <code class="docutils literal notranslate"><span class="pre">commitlog_segment_size_in_mb</span></code>, which itself default to 32MB).</p> |
| </div> |
| <div class="section" id="nodetool-says-connection-refused-to-host-127-0-1-1-for-any-remote-host-what-gives"> |
| <span id="nodetool-connection-refused"></span><h2>Nodetool says “Connection refused to host: 127.0.1.1” for any remote host. What gives?<a class="headerlink" href="#nodetool-says-connection-refused-to-host-127-0-1-1-for-any-remote-host-what-gives" title="Permalink to this headline">¶</a></h2> |
| <p>Nodetool relies on JMX, which in turn relies on RMI, which in turn sets up its own listeners and connectors as needed on |
| each end of the exchange. Normally all of this happens behind the scenes transparently, but incorrect name resolution |
| for either the host connecting, or the one being connected to, can result in crossed wires and confusing exceptions.</p> |
| <p>If you are not using DNS, then make sure that your <code class="docutils literal notranslate"><span class="pre">/etc/hosts</span></code> files are accurate on both ends. If that fails, try |
| setting the <code class="docutils literal notranslate"><span class="pre">-Djava.rmi.server.hostname=<public</span> <span class="pre">name></span></code> JVM option near the bottom of <code class="docutils literal notranslate"><span class="pre">cassandra-env.sh</span></code> to an |
| interface that you can reach from the remote machine.</p> |
| </div> |
| <div class="section" id="will-batching-my-operations-speed-up-my-bulk-load"> |
| <span id="to-batch-or-not-to-batch"></span><h2>Will batching my operations speed up my bulk load?<a class="headerlink" href="#will-batching-my-operations-speed-up-my-bulk-load" title="Permalink to this headline">¶</a></h2> |
| <p>No. Using batches to load data will generally just add “spikes” of latency. Use asynchronous INSERTs instead, or use |
| true <a class="reference internal" href="../operating/bulk_loading.html#bulk-loading"><span class="std std-ref">Bulk Loading</span></a>.</p> |
| <p>An exception is batching updates to a single partition, which can be a Good Thing (as long as the size of a single batch |
| stay reasonable). But never ever blindly batch everything!</p> |
| </div> |
| <div class="section" id="on-rhel-nodes-are-unable-to-join-the-ring"> |
| <span id="selinux"></span><h2>On RHEL nodes are unable to join the ring<a class="headerlink" href="#on-rhel-nodes-are-unable-to-join-the-ring" title="Permalink to this headline">¶</a></h2> |
| <p>Check if <a class="reference external" href="https://en.wikipedia.org/wiki/Security-Enhanced_Linux">SELinux</a> is on; if it is, turn it off.</p> |
| </div> |
| <div class="section" id="how-do-i-unsubscribe-from-the-email-list"> |
| <span id="how-to-unsubscribe"></span><h2>How do I unsubscribe from the email list?<a class="headerlink" href="#how-do-i-unsubscribe-from-the-email-list" title="Permalink to this headline">¶</a></h2> |
| <p>Send an email to <code class="docutils literal notranslate"><span class="pre">user-unsubscribe@cassandra.apache.org</span></code>.</p> |
| </div> |
| <div class="section" id="why-does-top-report-that-cassandra-is-using-a-lot-more-memory-than-the-java-heap-max"> |
| <span id="cassandra-eats-all-my-memory"></span><h2>Why does top report that Cassandra is using a lot more memory than the Java heap max?<a class="headerlink" href="#why-does-top-report-that-cassandra-is-using-a-lot-more-memory-than-the-java-heap-max" title="Permalink to this headline">¶</a></h2> |
| <p>Cassandra uses <a class="reference external" href="https://en.wikipedia.org/wiki/Memory-mapped_file">Memory Mapped Files</a> (mmap) internally. That is, we |
| use the operating system’s virtual memory system to map a number of on-disk files into the Cassandra process’ address |
| space. This will “use” virtual memory; i.e. address space, and will be reported by tools like top accordingly, but on 64 |
| bit systems virtual address space is effectively unlimited so you should not worry about that.</p> |
| <p>What matters from the perspective of “memory use” in the sense as it is normally meant, is the amount of data allocated |
| on brk() or mmap’d /dev/zero, which represent real memory used. The key issue is that for a mmap’d file, there is never |
| a need to retain the data resident in physical memory. Thus, whatever you do keep resident in physical memory is |
| essentially just there as a cache, in the same way as normal I/O will cause the kernel page cache to retain data that |
| you read/write.</p> |
| <p>The difference between normal I/O and mmap() is that in the mmap() case the memory is actually mapped to the process, |
| thus affecting the virtual size as reported by top. The main argument for using mmap() instead of standard I/O is the |
| fact that reading entails just touching memory - in the case of the memory being resident, you just read it - you don’t |
| even take a page fault (so no overhead in entering the kernel and doing a semi-context switch). This is covered in more |
| detail <a class="reference external" href="http://www.varnish-cache.org/trac/wiki/ArchitectNotes">here</a>.</p> |
| </div> |
| <div class="section" id="what-are-seeds"> |
| <span id="id1"></span><h2>What are seeds?<a class="headerlink" href="#what-are-seeds" title="Permalink to this headline">¶</a></h2> |
| <p>Seeds are used during startup to discover the cluster.</p> |
| <p>If you configure your nodes to refer some node as seed, nodes in your ring tend to send Gossip message to seeds more |
| often (also see the <a class="reference internal" href="../architecture/dynamo.html#gossip"><span class="std std-ref">section on gossip</span></a>) than to non-seeds. In other words, seeds are worked as hubs of |
| Gossip network. With seeds, each node can detect status changes of other nodes quickly.</p> |
| <p>Seeds are also referred by new nodes on bootstrap to learn other nodes in ring. When you add a new node to ring, you |
| need to specify at least one live seed to contact. Once a node join the ring, it learns about the other nodes, so it |
| doesn’t need seed on subsequent boot.</p> |
| <p>You can make a seed a node at any time. There is nothing special about seed nodes. If you list the node in seed list it |
| is a seed</p> |
| <p>Seeds do not auto bootstrap (i.e. if a node has itself in its seed list it will not automatically transfer data to itself) |
| If you want a node to do that, bootstrap it first and then add it to seeds later. If you have no data (new install) you |
| do not have to worry about bootstrap at all.</p> |
| <p>Recommended usage of seeds:</p> |
| <ul class="simple"> |
| <li>pick two (or more) nodes per data center as seed nodes.</li> |
| <li>sync the seed list to all your nodes</li> |
| </ul> |
| </div> |
| <div class="section" id="does-single-seed-mean-single-point-of-failure"> |
| <span id="are-seeds-spof"></span><h2>Does single seed mean single point of failure?<a class="headerlink" href="#does-single-seed-mean-single-point-of-failure" title="Permalink to this headline">¶</a></h2> |
| <p>The ring can operate or boot without a seed; however, you will not be able to add new nodes to the cluster. It is |
| recommended to configure multiple seeds in production system.</p> |
| </div> |
| <div class="section" id="why-can-t-i-call-jmx-method-x-on-jconsole"> |
| <span id="cant-call-jmx-method"></span><h2>Why can’t I call jmx method X on jconsole?<a class="headerlink" href="#why-can-t-i-call-jmx-method-x-on-jconsole" title="Permalink to this headline">¶</a></h2> |
| <p>Some of JMX operations use array argument and as jconsole doesn’t support array argument, those operations can’t be |
| called with jconsole (the buttons are inactive for them). You need to write a JMX client to call such operations or need |
| array-capable JMX monitoring tool.</p> |
| </div> |
| <div class="section" id="why-do-i-see-messages-dropped-in-the-logs"> |
| <span id="why-message-dropped"></span><h2>Why do I see “… messages dropped …” in the logs?<a class="headerlink" href="#why-do-i-see-messages-dropped-in-the-logs" title="Permalink to this headline">¶</a></h2> |
| <p>This is a symptom of load shedding – Cassandra defending itself against more requests than it can handle.</p> |
| <p>Internode messages which are received by a node, but do not get not to be processed within their proper timeout (see |
| <code class="docutils literal notranslate"><span class="pre">read_request_timeout</span></code>, <code class="docutils literal notranslate"><span class="pre">write_request_timeout</span></code>, … in the <a class="reference internal" href="../configuration/cassandra_config_file.html#cassandra-yaml"><span class="std std-ref">Cassandra Configuration File</span></a>), are dropped rather than |
| processed (since the as the coordinator node will no longer be waiting for a response).</p> |
| <p>For writes, this means that the mutation was not applied to all replicas it was sent to. The inconsistency will be |
| repaired by read repair, hints or a manual repair. The write operation may also have timeouted as a result.</p> |
| <p>For reads, this means a read request may not have completed.</p> |
| <p>Load shedding is part of the Cassandra architecture, if this is a persistent issue it is generally a sign of an |
| overloaded node or cluster.</p> |
| </div> |
| <div class="section" id="cassandra-dies-with-java-lang-outofmemoryerror-map-failed"> |
| <span id="oom-map-failed"></span><h2>Cassandra dies with <code class="docutils literal notranslate"><span class="pre">java.lang.OutOfMemoryError:</span> <span class="pre">Map</span> <span class="pre">failed</span></code><a class="headerlink" href="#cassandra-dies-with-java-lang-outofmemoryerror-map-failed" title="Permalink to this headline">¶</a></h2> |
| <p>If Cassandra is dying <strong>specifically</strong> with the “Map failed” message, it means the OS is denying java the ability to |
| lock more memory. In linux, this typically means memlock is limited. Check <code class="docutils literal notranslate"><span class="pre">/proc/<pid</span> <span class="pre">of</span> <span class="pre">cassandra>/limits</span></code> to verify |
| this and raise it (eg, via ulimit in bash). You may also need to increase <code class="docutils literal notranslate"><span class="pre">vm.max_map_count.</span></code> Note that the debian |
| package handles this for you automatically.</p> |
| </div> |
| <div class="section" id="what-happens-if-two-updates-are-made-with-the-same-timestamp"> |
| <span id="what-on-same-timestamp-update"></span><h2>What happens if two updates are made with the same timestamp?<a class="headerlink" href="#what-happens-if-two-updates-are-made-with-the-same-timestamp" title="Permalink to this headline">¶</a></h2> |
| <p>Updates must be commutative, since they may arrive in different orders on different replicas. As long as Cassandra has a |
| deterministic way to pick the winner (in a timestamp tie), the one selected is as valid as any other, and the specifics |
| should be treated as an implementation detail. That said, in the case of a timestamp tie, Cassandra follows two rules: |
| first, deletes take precedence over inserts/updates. Second, if there are two updates, the one with the lexically larger |
| value is selected.</p> |
| </div> |
| <div class="section" id="why-bootstrapping-a-new-node-fails-with-a-stream-failed-error"> |
| <span id="why-bootstrapping-stream-error"></span><h2>Why bootstrapping a new node fails with a “Stream failed” error?<a class="headerlink" href="#why-bootstrapping-a-new-node-fails-with-a-stream-failed-error" title="Permalink to this headline">¶</a></h2> |
| <p>Two main possibilities:</p> |
| <ol class="arabic simple"> |
| <li>the GC may be creating long pauses disrupting the streaming process</li> |
| <li>compactions happening in the background hold streaming long enough that the TCP connection fails</li> |
| </ol> |
| <p>In the first case, regular GC tuning advices apply. In the second case, you need to set TCP keepalive to a lower value |
| (default is very high on Linux). Try to just run the following:</p> |
| <div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ sudo /sbin/sysctl -w net.ipv4.tcp_keepalive_time=60 net.ipv4.tcp_keepalive_intvl=60 net.ipv4.tcp_keepalive_probes=5 |
| </pre></div> |
| </div> |
| <p>To make those settings permanent, add them to your <code class="docutils literal notranslate"><span class="pre">/etc/sysctl.conf</span></code> file.</p> |
| <p>Note: <a class="reference external" href="https://cloud.google.com/compute/">GCE</a>’s firewall will always interrupt TCP connections that are inactive for |
| more than 10 min. Running the above command is highly recommended in that environment.</p> |
| </div> |
| </div> |
| |
| |
| |
| |
| <div class="doc-prev-next-links" role="navigation" aria-label="footer navigation"> |
| |
| <a href="../plugins/index.html" class="btn btn-default pull-right " role="button" title="Third-Party Plugins" accesskey="n">Next <span class="glyphicon glyphicon-circle-arrow-right" aria-hidden="true"></span></a> |
| |
| |
| <a href="../development/release_process.html" class="btn btn-default" role="button" title="Release Process" accesskey="p"><span class="glyphicon glyphicon-circle-arrow-left" aria-hidden="true"></span> Previous</a> |
| |
| </div> |
| |
| </div> |
| </div> |
| </div> |
| </div> |
| </div> |