0102/streams/core-concepts.html - kafka-site - Git at Google

 <!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at

     http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
 -->

 <script><!--#include virtual="../js/templateData.js" --></script>

 <script id="content-template" type="text/x-handlebars-template">
     <h1>Core Concepts</h1>
     <p>
     We first summarize the key concepts of Kafka Streams.
     </p>

     <h3><a id="streams_topology" href="#streams_topology">Stream Processing Topology</a></h3>

     <ul>
         <li>A <b>stream</b> is the most important abstraction provided by Kafka Streams: it represents an unbounded, continuously updating data set. A stream is an ordered, replayable, and fault-tolerant sequence of immutable data records, where a <b>data record</b> is defined as a key-value pair.</li>
         <li>A <b>stream processing application</b> is any program that makes use of the Kafka Streams library. It defines its computational logic through one or more <b>processor topologies</b>, where a processor topology is a graph of stream processors (nodes) that are connected by streams (edges).</li>
         <li>A <b>stream processor</b> is a node in the processor topology; it represents a processing step to transform data in streams by receiving one input record at a time from its upstream processors in the topology, applying its operation to it, and may subsequently produce one or more output records to its downstream processors. </li>
     </ul>

     There are two special processors in the topology:

     <ul>
         <li><b>Source Processor</b>: A source processor is a special type of stream processor that does not have any upstream processors. It produces an input stream to its topology from one or multiple Kafka topics by consuming records from these topics and forward them to its down-stream processors.</li>
         <li><b>Sink Processor</b>: A sink processor is a special type of stream processor that does not have down-stream processors. It sends any received records from its up-stream processors to a specified Kafka topic.</li>
     </ul>

     <img class="centered" src="/{{version}}/images/streams-architecture-topology.jpg" style="width:400px">

     <p>
         Kafka Streams offers two ways to define the stream processing topology: the <a href="#streams_dsl"><b>Kafka Streams DSL</b></a> provides
         the most common data transformation operations such as <code>map</code>, <code>filter</code>, <code>join</code> and <code>aggregations</code> out of the box; the lower-level <a href="#streams_processor"><b>Processor API</b></a> allows
         developers define and connect custom processors as well as to interact with <a href="#streams_state">state stores</a>.
     </p>

     <p>
         A processor topology is merely a logical abstraction for your stream processing code.
         At runtime, the logical topology is instantiated and replicated inside the application for parallel processing (see <a href="#streams_architecture_tasks">Stream Partitions and Tasks</a> for details).
     </p>

     <h3><a id="streams_time" href="#streams_time">Time</a></h3>

     <p>
         A critical aspect in stream processing is the notion of <b>time</b>, and how it is modeled and integrated.
         For example, some operations such as <b>windowing</b> are defined based on time boundaries.
     </p>
     <p>
         Common notions of time in streams are:
     </p>

     <ul>
         <li><b>Event time</b> - The point in time when an event or data record occurred, i.e. was originally created "at the source". <b>Example:</b> If the event is a geo-location change reported by a GPS sensor in a car, then the associated event-time would be the time when the GPS sensor captured the location change.</li>
         <li><b>Processing time</b> - The point in time when the event or data record happens to be processed by the stream processing application, i.e. when the record is being consumed. The processing time may be milliseconds, hours, or days etc. later than the original event time. <b>Example:</b> Imagine an analytics application that reads and processes the geo-location data reported from car sensors to present it to a fleet management dashboard. Here, processing-time in the analytics application might be milliseconds or seconds (e.g. for real-time pipelines based on Apache Kafka and Kafka Streams) or hours (e.g. for batch pipelines based on Apache Hadoop or Apache Spark) after event-time.</li>
         <li><b>Ingestion time</b> - The point in time when an event or data record is stored in a topic partition by a Kafka broker. The difference to event time is that this ingestion timestamp is generated when the record is appended to the target topic by the Kafka broker, not when the record is created "at the source". The difference to processing time is that processing time is when the stream processing application processes the record. <b>For example,</b> if a record is never processed, there is no notion of processing time for it, but it still has an ingestion time.</li>
     </ul>
     <p>
         The choice between event-time and ingestion-time is actually done through the configuration of Kafka (not Kafka Streams): From Kafka 0.10.x onwards, timestamps are automatically embedded into Kafka messages. Depending on Kafka's configuration these timestamps represent event-time or ingestion-time. The respective Kafka configuration setting can be specified on the broker level or per topic. The default timestamp extractor in Kafka Streams will retrieve these embedded timestamps as-is. Hence, the effective time semantics of your application depend on the effective Kafka configuration for these embedded timestamps.
     </p>
     <p>
         Kafka Streams assigns a <b>timestamp</b> to every data record
         via the <code>TimestampExtractor</code> interface.
         Concrete implementations of this interface may retrieve or compute timestamps based on the actual contents of data records such as an embedded timestamp field
         to provide event-time semantics, or use any other approach such as returning the current wall-clock time at the time of processing,
         thereby yielding processing-time semantics to stream processing applications.
         Developers can thus enforce different notions of time depending on their business needs. For example,
         per-record timestamps describe the progress of a stream with regards to time (although records may be out-of-order within the stream) and
         are leveraged by time-dependent operations such as joins.
     </p>

     <p>
         Finally, whenever a Kafka Streams application writes records to Kafka, then it will also assign timestamps to these new records. The way the timestamps are assigned depends on the context:
     </p>

     <ul>
         <li> When new output records are generated via processing some input record, for example, <code>context.forward()</code> triggered in the <code>process()</code> function call, output record timestamps are inherited from input record timestamps directly.</li>
         <li> When new output records are generated via periodic functions such as <code>punctuate()</code>, the output record timestamp is defined as the current internal time (obtained through <code>context.timestamp()</code>) of the stream task.</li>
         <li> For aggregations, the timestamp of a resulting aggregate update record will be that of the latest arrived input record that triggered the update.</li>
     </ul>

     <h3><a id="streams_state" href="#streams_state">States</a></h3>

     <p>
         Some stream processing applications don't require state, which means the processing of a message is independent from
         the processing of all other messages.
         However, being able to maintain state opens up many possibilities for sophisticated stream processing applications: you
         can join input streams, or group and aggregate data records. Many such stateful operators are provided by the <a href="#streams_dsl"><b>Kafka Streams DSL</b></a>.
     </p>
     <p>
         Kafka Streams provides so-called <b>state stores</b>, which can be used by stream processing applications to store and query data.
         This is an important capability when implementing stateful operations.
         Every task in Kafka Streams embeds one or more state stores that can be accessed via APIs to store and query data required for processing.
         These state stores can either be a persistent key-value store, an in-memory hashmap, or another convenient data structure.
         Kafka Streams offers fault-tolerance and automatic recovery for local state stores.
     </p>
     <p>
         Kafka Streams allows direct read-only queries of the state stores by methods, threads, processes or applications external to the stream processing application that created the state stores. This is provided through a feature called <b>Interactive Queries</b>. All stores are named and Interactive Queries exposes only the read operations of the underlying implementation.
     </p>

     <div class="pagination">
         <a href="/{{version}}/documentation/streams" class="pagination__btn pagination__btn__prev">Previous</a>
         <a href="/{{version}}/documentation/streams/architecture" class="pagination__btn pagination__btn__next">Next</a>
     </div>
 </script>

 <!--#include virtual="../../includes/_header.htm" -->
 <!--#include virtual="../../includes/_top.htm" -->
 <div class="content documentation documentation--current">
 	<!--#include virtual="../../includes/_nav.htm" -->
 	<div class="right">
 		<!--#include virtual="../../includes/_docs_banner.htm" -->
         <ul class="breadcrumbs">
             <li><a href="/documentation">Documentation</a></li>
             <li><a href="/documentation/streams">Streams</a></li>
         </ul>
         <div class="p-content"></div>
     </div>
 </div>
 <!--#include virtual="../../includes/_footer.htm" -->
 <script>
 $(function() {
   // Show selected style on nav item
   $('.b-nav__streams').addClass('selected');

   // Display docs subnav items
   $('.b-nav__docs').parent().toggleClass('nav__item__with__subs--expanded');
 });
 </script>
	<!--
	Licensed to the Apache Software Foundation (ASF) under one or more
	contributor license agreements. See the NOTICE file distributed with
	this work for additional information regarding copyright ownership.
	The ASF licenses this file to You under the Apache License, Version 2.0
	(the "License"); you may not use this file except in compliance with
	the License. You may obtain a copy of the License at

	http://www.apache.org/licenses/LICENSE-2.0

	Unless required by applicable law or agreed to in writing, software
	distributed under the License is distributed on an "AS IS" BASIS,
	WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	See the License for the specific language governing permissions and
	limitations under the License.
	-->

	<script><!--#include virtual="../js/templateData.js" --></script>

	<script id="content-template" type="text/x-handlebars-template">
	<h1>Core Concepts</h1>
	<p>
	We first summarize the key concepts of Kafka Streams.
	</p>

	<h3><a id="streams_topology" href="#streams_topology">Stream Processing Topology</a></h3>

	<ul>
	<li>A <b>stream</b> is the most important abstraction provided by Kafka Streams: it represents an unbounded, continuously updating data set. A stream is an ordered, replayable, and fault-tolerant sequence of immutable data records, where a <b>data record</b> is defined as a key-value pair.</li>
	<li>A <b>stream processing application</b> is any program that makes use of the Kafka Streams library. It defines its computational logic through one or more <b>processor topologies</b>, where a processor topology is a graph of stream processors (nodes) that are connected by streams (edges).</li>
	<li>A <b>stream processor</b> is a node in the processor topology; it represents a processing step to transform data in streams by receiving one input record at a time from its upstream processors in the topology, applying its operation to it, and may subsequently produce one or more output records to its downstream processors. </li>
	</ul>

	There are two special processors in the topology:

	<ul>
	<li><b>Source Processor</b>: A source processor is a special type of stream processor that does not have any upstream processors. It produces an input stream to its topology from one or multiple Kafka topics by consuming records from these topics and forward them to its down-stream processors.</li>
	<li><b>Sink Processor</b>: A sink processor is a special type of stream processor that does not have down-stream processors. It sends any received records from its up-stream processors to a specified Kafka topic.</li>
	</ul>

	<img class="centered" src="/{{version}}/images/streams-architecture-topology.jpg" style="width:400px">

	<p>
	Kafka Streams offers two ways to define the stream processing topology: the <a href="#streams_dsl"><b>Kafka Streams DSL</b></a> provides
	the most common data transformation operations such as <code>map</code>, <code>filter</code>, <code>join</code> and <code>aggregations</code> out of the box; the lower-level <a href="#streams_processor"><b>Processor API</b></a> allows
	developers define and connect custom processors as well as to interact with <a href="#streams_state">state stores</a>.
	</p>

	<p>
	A processor topology is merely a logical abstraction for your stream processing code.
	At runtime, the logical topology is instantiated and replicated inside the application for parallel processing (see <a href="#streams_architecture_tasks">Stream Partitions and Tasks</a> for details).
	</p>

	<h3><a id="streams_time" href="#streams_time">Time</a></h3>

	<p>
	A critical aspect in stream processing is the notion of <b>time</b>, and how it is modeled and integrated.
	For example, some operations such as <b>windowing</b> are defined based on time boundaries.
	</p>
	<p>
	Common notions of time in streams are:
	</p>

	<ul>
	<li><b>Event time</b> - The point in time when an event or data record occurred, i.e. was originally created "at the source". <b>Example:</b> If the event is a geo-location change reported by a GPS sensor in a car, then the associated event-time would be the time when the GPS sensor captured the location change.</li>
	<li><b>Processing time</b> - The point in time when the event or data record happens to be processed by the stream processing application, i.e. when the record is being consumed. The processing time may be milliseconds, hours, or days etc. later than the original event time. <b>Example:</b> Imagine an analytics application that reads and processes the geo-location data reported from car sensors to present it to a fleet management dashboard. Here, processing-time in the analytics application might be milliseconds or seconds (e.g. for real-time pipelines based on Apache Kafka and Kafka Streams) or hours (e.g. for batch pipelines based on Apache Hadoop or Apache Spark) after event-time.</li>
	<li><b>Ingestion time</b> - The point in time when an event or data record is stored in a topic partition by a Kafka broker. The difference to event time is that this ingestion timestamp is generated when the record is appended to the target topic by the Kafka broker, not when the record is created "at the source". The difference to processing time is that processing time is when the stream processing application processes the record. <b>For example,</b> if a record is never processed, there is no notion of processing time for it, but it still has an ingestion time.</li>
	</ul>
	<p>
	The choice between event-time and ingestion-time is actually done through the configuration of Kafka (not Kafka Streams): From Kafka 0.10.x onwards, timestamps are automatically embedded into Kafka messages. Depending on Kafka's configuration these timestamps represent event-time or ingestion-time. The respective Kafka configuration setting can be specified on the broker level or per topic. The default timestamp extractor in Kafka Streams will retrieve these embedded timestamps as-is. Hence, the effective time semantics of your application depend on the effective Kafka configuration for these embedded timestamps.
	</p>
	<p>
	Kafka Streams assigns a <b>timestamp</b> to every data record
	via the <code>TimestampExtractor</code> interface.
	Concrete implementations of this interface may retrieve or compute timestamps based on the actual contents of data records such as an embedded timestamp field
	to provide event-time semantics, or use any other approach such as returning the current wall-clock time at the time of processing,
	thereby yielding processing-time semantics to stream processing applications.
	Developers can thus enforce different notions of time depending on their business needs. For example,
	per-record timestamps describe the progress of a stream with regards to time (although records may be out-of-order within the stream) and
	are leveraged by time-dependent operations such as joins.
	</p>

	<p>
	Finally, whenever a Kafka Streams application writes records to Kafka, then it will also assign timestamps to these new records. The way the timestamps are assigned depends on the context:
	</p>

	<ul>
	<li> When new output records are generated via processing some input record, for example, <code>context.forward()</code> triggered in the <code>process()</code> function call, output record timestamps are inherited from input record timestamps directly.</li>
	<li> When new output records are generated via periodic functions such as <code>punctuate()</code>, the output record timestamp is defined as the current internal time (obtained through <code>context.timestamp()</code>) of the stream task.</li>
	<li> For aggregations, the timestamp of a resulting aggregate update record will be that of the latest arrived input record that triggered the update.</li>
	</ul>

	<h3><a id="streams_state" href="#streams_state">States</a></h3>

	<p>
	Some stream processing applications don't require state, which means the processing of a message is independent from
	the processing of all other messages.
	However, being able to maintain state opens up many possibilities for sophisticated stream processing applications: you
	can join input streams, or group and aggregate data records. Many such stateful operators are provided by the <a href="#streams_dsl"><b>Kafka Streams DSL</b></a>.
	</p>
	<p>
	Kafka Streams provides so-called <b>state stores</b>, which can be used by stream processing applications to store and query data.
	This is an important capability when implementing stateful operations.
	Every task in Kafka Streams embeds one or more state stores that can be accessed via APIs to store and query data required for processing.
	These state stores can either be a persistent key-value store, an in-memory hashmap, or another convenient data structure.
	Kafka Streams offers fault-tolerance and automatic recovery for local state stores.
	</p>
	<p>
	Kafka Streams allows direct read-only queries of the state stores by methods, threads, processes or applications external to the stream processing application that created the state stores. This is provided through a feature called <b>Interactive Queries</b>. All stores are named and Interactive Queries exposes only the read operations of the underlying implementation.
	</p>

	<div class="pagination">
	<a href="/{{version}}/documentation/streams" class="pagination__btn pagination__btn__prev">Previous</a>
	<a href="/{{version}}/documentation/streams/architecture" class="pagination__btn pagination__btn__next">Next</a>
	</div>
	</script>

	<!--#include virtual="../../includes/_header.htm" -->
	<!--#include virtual="../../includes/_top.htm" -->
	<div class="content documentation documentation--current">
	<!--#include virtual="../../includes/_nav.htm" -->
	<div class="right">
	<!--#include virtual="../../includes/_docs_banner.htm" -->
	<ul class="breadcrumbs">
	<li><a href="/documentation">Documentation</a></li>
	<li><a href="/documentation/streams">Streams</a></li>
	</ul>
	<div class="p-content"></div>
	</div>
	</div>
	<!--#include virtual="../../includes/_footer.htm" -->
	<script>
	$(function() {
	// Show selected style on nav item
	$('.b-nav__streams').addClass('selected');

	// Display docs subnav items
	$('.b-nav__docs').parent().toggleClass('nav__item__with__subs--expanded');
	});
	</script>