_posts/2022-01-17-ignite1992.markdown - infrastructure-blogs-archive - Git at Google

 ---
 layout: post
 title: 'Apache Ignite 2.12.0: CDC, Index Query API, Vulnerabilities Fixes'
 date: '2022-01-17T13:46:58+00:00'
 categories: ignite
 ---
 <p>As of January 14, 2022, <a href="https://ignite.apache.org/">Apache Ignite</a> 2.12 has been released. You can directly check the full list of resolved <a href="https://s.apache.org/0zyi2">Important JIRA tasks</a> but here let’s briefly overview some valuable improvements.</p>
 <h3 id="vulnerability-updates">Vulnerability Updates</h3>
 <p>The Apache Ignite versions lower than 2.11.1 are vulnerable to <a href="https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44832">CVE-2021-44832</a> which is related to the <code>ignite-log4j2</code> module usage.</p>
 <p>The release also fixes 10+ CVE’s of various modules. See <a href="https://ignite.apache.org/releases/2.12.0/release_notes.html">release notes</a> for more details.</p>
 <h3 id="change-data-capture">Change Data Capture</h3>
 <p>Change Data Capture (<a href="https://en.wikipedia.org/wiki/Change_data_capture">CDC</a>) is a data processing pattern used to asynchronously receive entries that have been changed on the local node so that action can be taken using the changed entry.</p>
 <p>Below are some of the CDC use cases:</p>
 <ul>
 <li>Streaming changes in Warehouse;</li>
 <li>Updating search index;</li>
 <li>Calculating statistics (streaming queries);</li>
 <li>Auditing logs;</li>
 <li>Async interaction with external systems: Moderation, business process invocation, etc.</li>
 </ul>
 <p>Ignite implements CDC with the <code>ignite-cdc.sh</code> application and <a href="https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/cdc/CdcConsumer.java#L56">Java API</a>.</p>
 <p>Below are the CDC application and the Ignite node integrated via <a href="https://ignite.apache.org/docs/2.12.0/persistence/native-persistence#write-ahead-log">WAL archive segments</a>:</p>
 <p><img src="https://blogs.apache.org/ignite/mediaresource/c5574c7d-2a24-4def-b1c4-b2343ed98397" alt="Apache Ignite CDC design" style="display: block;margin-left: auto; margin-right: auto; width: 30%;"></p>
 <p>When CDC is enabled, the Ignite server node creates a hard link to each WAL archive segment in the special <code>db/cdc/{consistency_id}</code> directory. The <code>ignite-cdc.sh</code> application can be runruns on a different JVM and processes newly archived WAL segments. When the segment is fully processed by <code>ignite-cdc.sh</code>, it is removed. The actual disk space is free when both links (archive and CDC) are removed.</p>
 <p>State of consumption is a pointer to the last processed event. A consumer can tell <code>ignite-cdc.sh</code> to save the consumption state. On startup event processing will be continued from the last saved state.</p>
 <p>See implementation details <a href="https://ignite.apache.org/docs/2.12.0/persistence/change-data-capture">here</a>.</p>
 <h3 id="index-query-api">Index Query API</h3>
 <p>The Apache Ignite now provides Index Query API for existing indexes. Index queries work over distributed indexes and retrieve cache entries that match the specified query.</p>
 <p>It will help in some cases, where:</p>
 <ul>
 <li>SQL is not applicable by the design of user application;</li>
 <li><code>IndexScan</code> is preferable to <code>ScanQuery</code> for performance reasons.</li>
 </ul>
 <p>Example of query:</p>
 <pre><code class="lang-java"><span class="hljs-comment">// Find the persons who work in Organization 1 and have salary more than 1,000.</span>
 QueryCursor&lt;<span class="hljs-keyword">Cache</span>.Entry&lt;<span class="hljs-built_in">Integer</span>, Person&gt;&gt; cursor = <span class="hljs-keyword">cache</span>.query(
     <span class="hljs-literal">new</span> IndexQuery&lt;<span class="hljs-built_in">Integer</span>, Person&gt;(Person.class, <span class="hljs-string">"ORG_SALARY_IDX"</span>)
         .setCriteria(<span class="hljs-literal">eq</span>(<span class="hljs-string">"orgId"</span>, <span class="hljs-number">1</span>), <span class="hljs-literal">gt</span>(<span class="hljs-string">"salary"</span>, <span class="hljs-number">1000</span>))
 );
 </code></pre>
 <p>See more details <a href="https://ignite.apache.org/docs/latest/key-value-api/using-cache-queries#executing-index-queries">here</a>.</p>
 <h3 id="snapshots">Snapshots</h3>
 <p>Previous versions can perform snapshot restore in the same cluster topology only. The new version provides the ability to restore snapshots on different cluster topologies. Moreover, added support of encrypted caches.</p>
 <h3 id="distributed-environment-tests">Distributed Environment Tests</h3>
 <p>The <a href="https://cwiki.apache.org/confluence/display/IGNITE/IEP-56%3A+Distributed+environment+tests">new testing framework</a> was implemented. The main goal is to have a large enough set of integration tests that cover most of the typical cluster usage scenarios.</p>
 <p>Features:</p>
 <ul>
 <li>Ignite nodes can be started/stopped on a Docker or a real cluster with any custom configuration;</li>
 <li>Any Apache Ignite version is supported (released or compiled from sources);</li>
 <li> Apache Ignite forks are also supported «out of the box»;</li>
 <li>Any other application execution is also possible, e.g. we implemented starters for Spark and Zookeeper;</li>
 <li>The cluster can be managed using the <code>control.sh</code>, we made this a part of the test API;</li>
 <li>Custom Java applications can be executed remotely with/without a built-in Ignite node or a Thin client;</li>
 <li>Any ssh command can be executed remotely, and the result will be available locally (at the python test);</li>
 <li>A network can be broken by editing <code>iptables</code> to check communication issues;</li>
 <li>Tests can be executed in parallel when the cluster size is bigger than tests requirements.</li>
 </ul>
 <p>Framework based on <a href="https://ducktape-docs.readthedocs.io/en/latest/index.html">Ducktape</a> library from Kafka team, that&#39;s why we called it Ducktests.</p>
 <h3 id="migration-modules-to-the-apache-ignite-extensions">Migration modules to the Apache Ignite Extensions</h3>
 <p>There is activity on the migration of frameworks to extensions:</p>
 <ul>
 <li>GCE, AWS, Azure modules were migrated to <code>gce</code>, <code>aws</code>, <code>azure</code> extensions.</li>
 <li>CacheSpringStoreSessionListener was migrated to the <code>spring-tx</code> extension.</li>
 <li>TcpDiscoveryZookeeperIpFinder was migrated to the <code>zookeeper-ip-finder</code> extension.</li>
 </ul>
 <p>The binaries archive now weighs 10 percent less.</p>
 <h3 id="anything-else-">Anything else?</h3>
 <p>See the <a href="https://ignite.apache.org/releases/2.12.0/release_notes.html">release notes</a> to learn about all of the new features and bug fixes.</p>
 <p>Reach out to us on the community user list for more questions, details, and feedback.</p>
 <p>Sincerely yours,</p>
 <p>Ignite contributors and committers</p>
	---
	layout: post
	title: 'Apache Ignite 2.12.0: CDC, Index Query API, Vulnerabilities Fixes'
	date: '2022-01-17T13:46:58+00:00'
	categories: ignite
	---
	<p>As of January 14, 2022, <a href="https://ignite.apache.org/">Apache Ignite</a> 2.12 has been released. You can directly check the full list of resolved <a href="https://s.apache.org/0zyi2">Important JIRA tasks</a> but here let’s briefly overview some valuable improvements.</p>
	<h3 id="vulnerability-updates">Vulnerability Updates</h3>
	<p>The Apache Ignite versions lower than 2.11.1 are vulnerable to <a href="https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44832">CVE-2021-44832</a> which is related to the <code>ignite-log4j2</code> module usage.</p>
	<p>The release also fixes 10+ CVE’s of various modules. See <a href="https://ignite.apache.org/releases/2.12.0/release_notes.html">release notes</a> for more details.</p>
	<h3 id="change-data-capture">Change Data Capture</h3>
	<p>Change Data Capture (<a href="https://en.wikipedia.org/wiki/Change_data_capture">CDC</a>) is a data processing pattern used to asynchronously receive entries that have been changed on the local node so that action can be taken using the changed entry.</p>
	<p>Below are some of the CDC use cases:</p>
	<ul>
	<li>Streaming changes in Warehouse;</li>
	<li>Updating search index;</li>
	<li>Calculating statistics (streaming queries);</li>
	<li>Auditing logs;</li>
	<li>Async interaction with external systems: Moderation, business process invocation, etc.</li>
	</ul>
	<p>Ignite implements CDC with the <code>ignite-cdc.sh</code> application and <a href="https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/cdc/CdcConsumer.java#L56">Java API</a>.</p>
	<p>Below are the CDC application and the Ignite node integrated via <a href="https://ignite.apache.org/docs/2.12.0/persistence/native-persistence#write-ahead-log">WAL archive segments</a>:</p>
	<p><img src="https://blogs.apache.org/ignite/mediaresource/c5574c7d-2a24-4def-b1c4-b2343ed98397" alt="Apache Ignite CDC design" style="display: block;margin-left: auto; margin-right: auto; width: 30%;"></p>
	<p>When CDC is enabled, the Ignite server node creates a hard link to each WAL archive segment in the special <code>db/cdc/{consistency_id}</code> directory. The <code>ignite-cdc.sh</code> application can be runruns on a different JVM and processes newly archived WAL segments. When the segment is fully processed by <code>ignite-cdc.sh</code>, it is removed. The actual disk space is free when both links (archive and CDC) are removed.</p>
	<p>State of consumption is a pointer to the last processed event. A consumer can tell <code>ignite-cdc.sh</code> to save the consumption state. On startup event processing will be continued from the last saved state.</p>
	<p>See implementation details <a href="https://ignite.apache.org/docs/2.12.0/persistence/change-data-capture">here</a>.</p>
	<h3 id="index-query-api">Index Query API</h3>
	<p>The Apache Ignite now provides Index Query API for existing indexes. Index queries work over distributed indexes and retrieve cache entries that match the specified query.</p>
	<p>It will help in some cases, where:</p>
	<ul>
	<li>SQL is not applicable by the design of user application;</li>
	<li><code>IndexScan</code> is preferable to <code>ScanQuery</code> for performance reasons.</li>
	</ul>
	<p>Example of query:</p>
	<pre><code class="lang-java"><span class="hljs-comment">// Find the persons who work in Organization 1 and have salary more than 1,000.</span>
	QueryCursor<<span class="hljs-keyword">Cache</span>.Entry<<span class="hljs-built_in">Integer</span>, Person>> cursor = <span class="hljs-keyword">cache</span>.query(
	<span class="hljs-literal">new</span> IndexQuery<<span class="hljs-built_in">Integer</span>, Person>(Person.class, <span class="hljs-string">"ORG_SALARY_IDX"</span>)
	.setCriteria(<span class="hljs-literal">eq</span>(<span class="hljs-string">"orgId"</span>, <span class="hljs-number">1</span>), <span class="hljs-literal">gt</span>(<span class="hljs-string">"salary"</span>, <span class="hljs-number">1000</span>))
	);
	</code></pre>
	<p>See more details <a href="https://ignite.apache.org/docs/latest/key-value-api/using-cache-queries#executing-index-queries">here</a>.</p>
	<h3 id="snapshots">Snapshots</h3>
	<p>Previous versions can perform snapshot restore in the same cluster topology only. The new version provides the ability to restore snapshots on different cluster topologies. Moreover, added support of encrypted caches.</p>
	<h3 id="distributed-environment-tests">Distributed Environment Tests</h3>
	<p>The <a href="https://cwiki.apache.org/confluence/display/IGNITE/IEP-56%3A+Distributed+environment+tests">new testing framework</a> was implemented. The main goal is to have a large enough set of integration tests that cover most of the typical cluster usage scenarios.</p>
	<p>Features:</p>
	<ul>
	<li>Ignite nodes can be started/stopped on a Docker or a real cluster with any custom configuration;</li>
	<li>Any Apache Ignite version is supported (released or compiled from sources);</li>
	<li> Apache Ignite forks are also supported «out of the box»;</li>
	<li>Any other application execution is also possible, e.g. we implemented starters for Spark and Zookeeper;</li>
	<li>The cluster can be managed using the <code>control.sh</code>, we made this a part of the test API;</li>
	<li>Custom Java applications can be executed remotely with/without a built-in Ignite node or a Thin client;</li>
	<li>Any ssh command can be executed remotely, and the result will be available locally (at the python test);</li>
	<li>A network can be broken by editing <code>iptables</code> to check communication issues;</li>
	<li>Tests can be executed in parallel when the cluster size is bigger than tests requirements.</li>
	</ul>
	<p>Framework based on <a href="https://ducktape-docs.readthedocs.io/en/latest/index.html">Ducktape</a> library from Kafka team, that's why we called it Ducktests.</p>
	<h3 id="migration-modules-to-the-apache-ignite-extensions">Migration modules to the Apache Ignite Extensions</h3>
	<p>There is activity on the migration of frameworks to extensions:</p>
	<ul>
	<li>GCE, AWS, Azure modules were migrated to <code>gce</code>, <code>aws</code>, <code>azure</code> extensions.</li>
	<li>CacheSpringStoreSessionListener was migrated to the <code>spring-tx</code> extension.</li>
	<li>TcpDiscoveryZookeeperIpFinder was migrated to the <code>zookeeper-ip-finder</code> extension.</li>
	</ul>
	<p>The binaries archive now weighs 10 percent less.</p>
	<h3 id="anything-else-">Anything else?</h3>
	<p>See the <a href="https://ignite.apache.org/releases/2.12.0/release_notes.html">release notes</a> to learn about all of the new features and bug fixes.</p>
	<p>Reach out to us on the community user list for more questions, details, and feedback.</p>
	<p>Sincerely yours,</p>
	<p>Ignite contributors and committers</p>