docs/build/plain-html/topics/impala_performance.html - impala - Git at Google

 <?xml version="1.0" encoding="UTF-8"?>
 <!DOCTYPE html
   PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />

 <meta name="copyright" content="(C) Copyright 2025" />
 <meta name="DC.rights.owner" content="(C) Copyright 2025" />
 <meta name="DC.Type" content="concept" />
 <meta name="DC.Title" content="Tuning Impala for Performance" />
 <meta name="DC.Relation" scheme="URI" content="../topics/impala_perf_cookbook.html" />
 <meta name="DC.Relation" scheme="URI" content="../topics/impala_perf_joins.html" />
 <meta name="DC.Relation" scheme="URI" content="../topics/impala_perf_stats.html" />
 <meta name="DC.Relation" scheme="URI" content="../topics/impala_perf_benchmarking.html" />
 <meta name="DC.Relation" scheme="URI" content="../topics/impala_perf_resources.html" />
 <meta name="DC.Relation" scheme="URI" content="../topics/impala_runtime_filtering.html" />
 <meta name="DC.Relation" scheme="URI" content="../topics/impala_perf_hdfs_caching.html" />
 <meta name="DC.Relation" scheme="URI" content="../topics/impala_perf_skew.html" />
 <meta name="DC.Relation" scheme="URI" content="../topics/impala_data_cache.html" />
 <meta name="DC.Relation" scheme="URI" content="../topics/impala_perf_testing.html" />
 <meta name="DC.Relation" scheme="URI" content="../topics/impala_explain_plan.html" />
 <meta name="prodname" content="Impala" />
 <meta name="prodname" content="Impala" />
 <meta name="version" content="Impala 3.4.x" />
 <meta name="version" content="Impala 3.4.x" />
 <meta name="DC.Format" content="XHTML" />
 <meta name="DC.Identifier" content="performance" />
 <link rel="stylesheet" type="text/css" href="../commonltr.css" />
 <title>Tuning Impala for Performance</title>
 </head>
 <body id="performance">


   <h1 class="title topictitle1" id="ariaid-title1">Tuning Impala for Performance</h1>


   <div class="body conbody">

     <p class="p">
       The following sections explain the factors affecting the performance of Impala features, and procedures for
       tuning, monitoring, and benchmarking Impala queries and other SQL operations.
     </p>


     <p class="p">
       This section also describes techniques for maximizing Impala scalability. Scalability is tied to performance:
       it means that performance remains high as the system workload increases. For example, reducing the disk I/O
       performed by a query can speed up an individual query, and at the same time improve scalability by making it
       practical to run more queries simultaneously. Sometimes, an optimization technique improves scalability more
       than performance. For example, reducing memory usage for a query might not change the query performance much,
       but might improve scalability by allowing more Impala queries or other kinds of jobs to run at the same time
       without running out of memory.
     </p>


     <div class="note note"><span class="notetitle">Note:</span>
       <p class="p">
         Before starting any performance tuning or benchmarking, make sure your system is configured with all the
         recommended minimum hardware requirements from <a class="xref" href="impala_prereqs.html#prereqs_hardware">Hardware Requirements</a> and
         software settings from <a class="xref" href="impala_config_performance.html#config_performance">Post-Installation Configuration for Impala</a>.
       </p>

     </div>


     <ul class="ul">
       <li class="li">
         <a class="xref" href="impala_partitioning.html#partitioning">Partitioning for Impala Tables</a>. This technique physically divides the data based on
         the different values in frequently queried columns, allowing queries to skip reading a large percentage of
         the data in a table.
       </li>


       <li class="li">
         <a class="xref" href="impala_perf_joins.html#perf_joins">Performance Considerations for Join Queries</a>. Joins are the main class of queries that you can tune at
         the SQL level, as opposed to changing physical factors such as the file format or the hardware
         configuration. The related topics <a class="xref" href="impala_perf_stats.html#perf_column_stats">Overview of Column Statistics</a> and
         <a class="xref" href="impala_perf_stats.html#perf_table_stats">Overview of Table Statistics</a> are also important primarily for join performance.
       </li>


       <li class="li">
         <a class="xref" href="impala_perf_stats.html#perf_table_stats">Overview of Table Statistics</a> and
         <a class="xref" href="impala_perf_stats.html#perf_column_stats">Overview of Column Statistics</a>. Gathering table and column statistics, using the
         <code class="ph codeph">COMPUTE STATS</code> statement, helps Impala automatically optimize the performance for join
         queries, without requiring changes to SQL query statements. (This process is greatly simplified in Impala
         1.2.2 and higher, because the <code class="ph codeph">COMPUTE STATS</code> statement gathers both kinds of statistics in
         one operation, and does not require any setup and configuration as was previously necessary for the
         <code class="ph codeph">ANALYZE TABLE</code> statement in Hive.)
       </li>


       <li class="li">
         <a class="xref" href="impala_perf_testing.html#performance_testing">Testing Impala Performance</a>. Do some post-setup testing to ensure Impala is
         using optimal settings for performance, before conducting any benchmark tests.
       </li>


       <li class="li">
         <a class="xref" href="impala_perf_benchmarking.html#perf_benchmarks">Benchmarking Impala Queries</a>. The configuration and sample data that you use
         for initial experiments with Impala is often not appropriate for doing performance tests.
       </li>


       <li class="li">
         <a class="xref" href="impala_perf_resources.html#mem_limits">Controlling Impala Resource Usage</a>. The more memory Impala can utilize, the better query
         performance you can expect. In a cluster running other kinds of workloads as well, you must make tradeoffs
         to make sure all Hadoop components have enough memory to perform well, so you might cap the memory that
         Impala can use.
       </li>


       <li class="li">
         <a class="xref" href="impala_s3.html#s3">Using Impala with Amazon S3 Object Store</a>. Queries against data stored in the Amazon Simple Storage Service (S3)
         have different performance characteristics than when the data is stored in HDFS.
       </li>

     </ul>


     <p class="p toc"></p>


     <p class="p">
         A good source of tips related to scalability and performance tuning is the
         <a class="xref" href="http://www.slideshare.net/cloudera/the-impala-cookbook-42530186" target="_blank">Impala
         Cookbook</a> presentation. These slides are updated periodically as new features come
         out and new benchmarks are performed.
       </p>


   </div>


 <div class="related-links">
 <ul class="ullinks">
 <li class="link ulchildlink"><strong><a href="../topics/impala_perf_cookbook.html">Impala Performance Guidelines and Best Practices</a></strong><br />
 </li>
 <li class="link ulchildlink"><strong><a href="../topics/impala_perf_joins.html">Performance Considerations for Join Queries</a></strong><br />
 </li>
 <li class="link ulchildlink"><strong><a href="../topics/impala_perf_stats.html">Table and Column Statistics</a></strong><br />
 </li>
 <li class="link ulchildlink"><strong><a href="../topics/impala_perf_benchmarking.html">Benchmarking Impala Queries</a></strong><br />
 </li>
 <li class="link ulchildlink"><strong><a href="../topics/impala_perf_resources.html">Controlling Impala Resource Usage</a></strong><br />
 </li>
 <li class="link ulchildlink"><strong><a href="../topics/impala_runtime_filtering.html">Runtime Filtering for Impala Queries (Impala 2.5 or higher only)</a></strong><br />
 </li>
 <li class="link ulchildlink"><strong><a href="../topics/impala_perf_hdfs_caching.html">Using HDFS Caching with Impala (Impala 2.1 or higher only)</a></strong><br />
 </li>
 <li class="link ulchildlink"><strong><a href="../topics/impala_perf_skew.html">Detecting and Correcting HDFS Block Skew Conditions</a></strong><br />
 </li>
 <li class="link ulchildlink"><strong><a href="../topics/impala_data_cache.html">Data Cache for Remote Reads</a></strong><br />
 </li>
 <li class="link ulchildlink"><strong><a href="../topics/impala_perf_testing.html">Testing Impala Performance</a></strong><br />
 </li>
 <li class="link ulchildlink"><strong><a href="../topics/impala_explain_plan.html">Understanding Impala Query Performance - EXPLAIN Plans and Query Profiles</a></strong><br />
 </li>
 </ul>
 </div></body>
 </html>
	<?xml version="1.0" encoding="UTF-8"?>
	<!DOCTYPE html
	PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
	<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
	<head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />

	<meta name="copyright" content="(C) Copyright 2025" />
	<meta name="DC.rights.owner" content="(C) Copyright 2025" />
	<meta name="DC.Type" content="concept" />
	<meta name="DC.Title" content="Tuning Impala for Performance" />
	<meta name="DC.Relation" scheme="URI" content="../topics/impala_perf_cookbook.html" />
	<meta name="DC.Relation" scheme="URI" content="../topics/impala_perf_joins.html" />
	<meta name="DC.Relation" scheme="URI" content="../topics/impala_perf_stats.html" />
	<meta name="DC.Relation" scheme="URI" content="../topics/impala_perf_benchmarking.html" />
	<meta name="DC.Relation" scheme="URI" content="../topics/impala_perf_resources.html" />
	<meta name="DC.Relation" scheme="URI" content="../topics/impala_runtime_filtering.html" />
	<meta name="DC.Relation" scheme="URI" content="../topics/impala_perf_hdfs_caching.html" />
	<meta name="DC.Relation" scheme="URI" content="../topics/impala_perf_skew.html" />
	<meta name="DC.Relation" scheme="URI" content="../topics/impala_data_cache.html" />
	<meta name="DC.Relation" scheme="URI" content="../topics/impala_perf_testing.html" />
	<meta name="DC.Relation" scheme="URI" content="../topics/impala_explain_plan.html" />
	<meta name="prodname" content="Impala" />
	<meta name="prodname" content="Impala" />
	<meta name="version" content="Impala 3.4.x" />
	<meta name="version" content="Impala 3.4.x" />
	<meta name="DC.Format" content="XHTML" />
	<meta name="DC.Identifier" content="performance" />
	<link rel="stylesheet" type="text/css" href="../commonltr.css" />
	<title>Tuning Impala for Performance</title>
	</head>
	<body id="performance">


	<h1 class="title topictitle1" id="ariaid-title1">Tuning Impala for Performance</h1>




	<div class="body conbody">

	<p class="p">
	The following sections explain the factors affecting the performance of Impala features, and procedures for
	tuning, monitoring, and benchmarking Impala queries and other SQL operations.
	</p>


	<p class="p">
	This section also describes techniques for maximizing Impala scalability. Scalability is tied to performance:
	it means that performance remains high as the system workload increases. For example, reducing the disk I/O
	performed by a query can speed up an individual query, and at the same time improve scalability by making it
	practical to run more queries simultaneously. Sometimes, an optimization technique improves scalability more
	than performance. For example, reducing memory usage for a query might not change the query performance much,
	but might improve scalability by allowing more Impala queries or other kinds of jobs to run at the same time
	without running out of memory.
	</p>


	<div class="note note"><span class="notetitle">Note:</span>
	<p class="p">
	Before starting any performance tuning or benchmarking, make sure your system is configured with all the
	recommended minimum hardware requirements from <a class="xref" href="impala_prereqs.html#prereqs_hardware">Hardware Requirements</a> and
	software settings from <a class="xref" href="impala_config_performance.html#config_performance">Post-Installation Configuration for Impala</a>.
	</p>

	</div>


	<ul class="ul">
	<li class="li">
	<a class="xref" href="impala_partitioning.html#partitioning">Partitioning for Impala Tables</a>. This technique physically divides the data based on
	the different values in frequently queried columns, allowing queries to skip reading a large percentage of
	the data in a table.
	</li>


	<li class="li">
	<a class="xref" href="impala_perf_joins.html#perf_joins">Performance Considerations for Join Queries</a>. Joins are the main class of queries that you can tune at
	the SQL level, as opposed to changing physical factors such as the file format or the hardware
	configuration. The related topics <a class="xref" href="impala_perf_stats.html#perf_column_stats">Overview of Column Statistics</a> and
	<a class="xref" href="impala_perf_stats.html#perf_table_stats">Overview of Table Statistics</a> are also important primarily for join performance.
	</li>


	<li class="li">
	<a class="xref" href="impala_perf_stats.html#perf_table_stats">Overview of Table Statistics</a> and
	<a class="xref" href="impala_perf_stats.html#perf_column_stats">Overview of Column Statistics</a>. Gathering table and column statistics, using the
	<code class="ph codeph">COMPUTE STATS</code> statement, helps Impala automatically optimize the performance for join
	queries, without requiring changes to SQL query statements. (This process is greatly simplified in Impala
	1.2.2 and higher, because the <code class="ph codeph">COMPUTE STATS</code> statement gathers both kinds of statistics in
	one operation, and does not require any setup and configuration as was previously necessary for the
	<code class="ph codeph">ANALYZE TABLE</code> statement in Hive.)
	</li>


	<li class="li">
	<a class="xref" href="impala_perf_testing.html#performance_testing">Testing Impala Performance</a>. Do some post-setup testing to ensure Impala is
	using optimal settings for performance, before conducting any benchmark tests.
	</li>


	<li class="li">
	<a class="xref" href="impala_perf_benchmarking.html#perf_benchmarks">Benchmarking Impala Queries</a>. The configuration and sample data that you use
	for initial experiments with Impala is often not appropriate for doing performance tests.
	</li>


	<li class="li">
	<a class="xref" href="impala_perf_resources.html#mem_limits">Controlling Impala Resource Usage</a>. The more memory Impala can utilize, the better query
	performance you can expect. In a cluster running other kinds of workloads as well, you must make tradeoffs
	to make sure all Hadoop components have enough memory to perform well, so you might cap the memory that
	Impala can use.
	</li>




	<li class="li">
	<a class="xref" href="impala_s3.html#s3">Using Impala with Amazon S3 Object Store</a>. Queries against data stored in the Amazon Simple Storage Service (S3)
	have different performance characteristics than when the data is stored in HDFS.
	</li>

	</ul>


	<p class="p toc"></p>


	<p class="p">
	A good source of tips related to scalability and performance tuning is the
	<a class="xref" href="http://www.slideshare.net/cloudera/the-impala-cookbook-42530186" target="_blank">Impala
	Cookbook</a> presentation. These slides are updated periodically as new features come
	out and new benchmarks are performed.
	</p>


	</div>

























	<div class="related-links">
	<ul class="ullinks">
	<li class="link ulchildlink"><strong><a href="../topics/impala_perf_cookbook.html">Impala Performance Guidelines and Best Practices</a></strong><br />
	</li>
	<li class="link ulchildlink"><strong><a href="../topics/impala_perf_joins.html">Performance Considerations for Join Queries</a></strong><br />
	</li>
	<li class="link ulchildlink"><strong><a href="../topics/impala_perf_stats.html">Table and Column Statistics</a></strong><br />
	</li>
	<li class="link ulchildlink"><strong><a href="../topics/impala_perf_benchmarking.html">Benchmarking Impala Queries</a></strong><br />
	</li>
	<li class="link ulchildlink"><strong><a href="../topics/impala_perf_resources.html">Controlling Impala Resource Usage</a></strong><br />
	</li>
	<li class="link ulchildlink"><strong><a href="../topics/impala_runtime_filtering.html">Runtime Filtering for Impala Queries (Impala 2.5 or higher only)</a></strong><br />
	</li>
	<li class="link ulchildlink"><strong><a href="../topics/impala_perf_hdfs_caching.html">Using HDFS Caching with Impala (Impala 2.1 or higher only)</a></strong><br />
	</li>
	<li class="link ulchildlink"><strong><a href="../topics/impala_perf_skew.html">Detecting and Correcting HDFS Block Skew Conditions</a></strong><br />
	</li>
	<li class="link ulchildlink"><strong><a href="../topics/impala_data_cache.html">Data Cache for Remote Reads</a></strong><br />
	</li>
	<li class="link ulchildlink"><strong><a href="../topics/impala_perf_testing.html">Testing Impala Performance</a></strong><br />
	</li>
	<li class="link ulchildlink"><strong><a href="../topics/impala_explain_plan.html">Understanding Impala Query Performance - EXPLAIN Plans and Query Profiles</a></strong><br />
	</li>
	</ul>
	</div></body>
	</html>