| <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
| <!-- |
| | Generated by Apache Maven Doxia at 2021-06-15 |
| | Rendered using Apache Maven Stylus Skin 1.5 |
| --> |
| <html xmlns="http://www.w3.org/1999/xhtml"> |
| <head> |
| <title>Apache Hadoop 3.3.1 – Statistic collection with the IOStatistics API</title> |
| <style type="text/css" media="all"> |
| @import url("../css/maven-base.css"); |
| @import url("../css/maven-theme.css"); |
| @import url("../css/site.css"); |
| </style> |
| <link rel="stylesheet" href="../css/print.css" type="text/css" media="print" /> |
| <meta name="Date-Revision-yyyymmdd" content="20210615" /> |
| <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> |
| </head> |
| <body class="composite"> |
| <div id="banner"> |
| <a href="http://hadoop.apache.org/" id="bannerLeft"> |
| <img src="http://hadoop.apache.org/images/hadoop-logo.jpg" alt="" /> |
| </a> |
| <a href="http://www.apache.org/" id="bannerRight"> |
| <img src="http://www.apache.org/images/asf_logo_wide.png" alt="" /> |
| </a> |
| <div class="clear"> |
| <hr/> |
| </div> |
| </div> |
| <div id="breadcrumbs"> |
| |
| <div class="xleft"> |
| <a href="http://www.apache.org/" class="externalLink">Apache</a> |
| > |
| <a href="http://hadoop.apache.org/" class="externalLink">Hadoop</a> |
| > |
| <a href="../../index.html">Apache Hadoop Project Dist POM</a> |
| > |
| <a href="../index.html">Apache Hadoop 3.3.1</a> |
| > |
| Statistic collection with the IOStatistics API |
| </div> |
| <div class="xright"> <a href="http://wiki.apache.org/hadoop" class="externalLink">Wiki</a> |
| | |
| <a href="https://gitbox.apache.org/repos/asf/hadoop.git" class="externalLink">git</a> |
| | |
| <a href="http://hadoop.apache.org/" class="externalLink">Apache Hadoop</a> |
| |
| | Last Published: 2021-06-15 |
| | Version: 3.3.1 |
| </div> |
| <div class="clear"> |
| <hr/> |
| </div> |
| </div> |
| <div id="leftColumn"> |
| <div id="navcolumn"> |
| |
| <h5>General</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../index.html">Overview</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/SingleCluster.html">Single Node Setup</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/ClusterSetup.html">Cluster Setup</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/CommandsManual.html">Commands Reference</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/FileSystemShell.html">FileSystem Shell</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/Compatibility.html">Compatibility Specification</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/DownstreamDev.html">Downstream Developer's Guide</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/AdminCompatibilityGuide.html">Admin Compatibility Guide</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/InterfaceClassification.html">Interface Classification</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/filesystem/index.html">FileSystem Specification</a> |
| </li> |
| </ul> |
| <h5>Common</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/CLIMiniCluster.html">CLI Mini Cluster</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/FairCallQueue.html">Fair Call Queue</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/NativeLibraries.html">Native Libraries</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/Superusers.html">Proxy User</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/RackAwareness.html">Rack Awareness</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/SecureMode.html">Secure Mode</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/ServiceLevelAuth.html">Service Level Authorization</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/HttpAuthentication.html">HTTP Authentication</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/CredentialProviderAPI.html">Credential Provider API</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-kms/index.html">Hadoop KMS</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/Tracing.html">Tracing</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/UnixShellGuide.html">Unix Shell Guide</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/registry/index.html">Registry</a> |
| </li> |
| </ul> |
| <h5>HDFS</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsDesign.html">Architecture</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html">User Guide</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HDFSCommands.html">Commands Reference</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html">NameNode HA With QJM</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithNFS.html">NameNode HA With NFS</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/ObserverNameNode.html">Observer NameNode</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/Federation.html">Federation</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/ViewFs.html">ViewFs</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/ViewFsOverloadScheme.html">ViewFsOverloadScheme</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsSnapshots.html">Snapshots</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsEditsViewer.html">Edits Viewer</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsImageViewer.html">Image Viewer</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html">Permissions and HDFS</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html">Quotas and HDFS</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/LibHdfs.html">libhdfs (C API)</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/WebHDFS.html">WebHDFS (REST API)</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-hdfs-httpfs/index.html">HttpFS</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html">Short Circuit Local Reads</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/CentralizedCacheManagement.html">Centralized Cache Management</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html">NFS Gateway</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html">Rolling Upgrade</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/ExtendedAttributes.html">Extended Attributes</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html">Transparent Encryption</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsMultihoming.html">Multihoming</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html">Storage Policies</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/MemoryStorage.html">Memory Storage Support</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/SLGUserGuide.html">Synthetic Load Generator</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html">Erasure Coding</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HDFSDiskbalancer.html">Disk Balancer</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsUpgradeDomain.html">Upgrade Domain</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsDataNodeAdminGuide.html">DataNode Admin</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs-rbf/HDFSRouterFederation.html">Router Federation</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsProvidedStorage.html">Provided Storage</a> |
| </li> |
| </ul> |
| <h5>MapReduce</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html">Tutorial</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredCommands.html">Commands Reference</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html">Compatibility with 1.x</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html">Encrypted Shuffle</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html">Pluggable Shuffle/Sort</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/DistributedCacheDeploy.html">Distributed Cache Deploy</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/SharedCacheSupport.html">Support for YARN Shared Cache</a> |
| </li> |
| </ul> |
| <h5>MapReduce REST APIs</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredAppMasterRest.html">MR Application Master</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-hs/HistoryServerRest.html">MR History Server</a> |
| </li> |
| </ul> |
| <h5>YARN</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/YARN.html">Architecture</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/YarnCommands.html">Commands Reference</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html">Capacity Scheduler</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/FairScheduler.html">Fair Scheduler</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html">ResourceManager Restart</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html">ResourceManager HA</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/ResourceModel.html">Resource Model</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/NodeLabel.html">Node Labels</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/NodeAttributes.html">Node Attributes</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/WebApplicationProxy.html">Web Application Proxy</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/TimelineServer.html">Timeline Server</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html">Timeline Service V.2</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html">Writing YARN Applications</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html">YARN Application Security</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/NodeManager.html">NodeManager</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/DockerContainers.html">Running Applications in Docker Containers</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/RuncContainers.html">Running Applications in runC Containers</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/NodeManagerCgroups.html">Using CGroups</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/SecureContainer.html">Secure Containers</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/ReservationSystem.html">Reservation System</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/GracefulDecommission.html">Graceful Decommission</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html">Opportunistic Containers</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/Federation.html">YARN Federation</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/SharedCache.html">Shared Cache</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/UsingGpus.html">Using GPU</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/UsingFPGA.html">Using FPGA</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html">Placement Constraints</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/YarnUI2.html">YARN UI2</a> |
| </li> |
| </ul> |
| <h5>YARN REST APIs</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html">Introduction</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html">Resource Manager</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/NodeManagerRest.html">Node Manager</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/TimelineServer.html#Timeline_Server_REST_API_v1">Timeline Server</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html#Timeline_Service_v.2_REST_API">Timeline Service V.2</a> |
| </li> |
| </ul> |
| <h5>YARN Service</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/yarn-service/Overview.html">Overview</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/yarn-service/QuickStart.html">QuickStart</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/yarn-service/Concepts.html">Concepts</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/yarn-service/YarnServiceAPI.html">Yarn Service API</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/yarn-service/ServiceDiscovery.html">Service Discovery</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/yarn-service/SystemServices.html">System Services</a> |
| </li> |
| </ul> |
| <h5>Hadoop Compatible File Systems</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-aliyun/tools/hadoop-aliyun/index.html">Aliyun OSS</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-aws/tools/hadoop-aws/index.html">Amazon S3</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-azure/index.html">Azure Blob Storage</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-azure-datalake/index.html">Azure Data Lake Storage</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-openstack/index.html">OpenStack Swift</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-cos/cloud-storage/index.html">Tencent COS</a> |
| </li> |
| </ul> |
| <h5>Auth</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-auth/index.html">Overview</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-auth/Examples.html">Examples</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-auth/Configuration.html">Configuration</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-auth/BuildingIt.html">Building</a> |
| </li> |
| </ul> |
| <h5>Tools</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-streaming/HadoopStreaming.html">Hadoop Streaming</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-archives/HadoopArchives.html">Hadoop Archives</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-archive-logs/HadoopArchiveLogs.html">Hadoop Archive Logs</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-distcp/DistCp.html">DistCp</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-gridmix/GridMix.html">GridMix</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-rumen/Rumen.html">Rumen</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-resourceestimator/ResourceEstimator.html">Resource Estimator Service</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-sls/SchedulerLoadSimulator.html">Scheduler Load Simulator</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/Benchmarking.html">Hadoop Benchmarking</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-dynamometer/Dynamometer.html">Dynamometer</a> |
| </li> |
| </ul> |
| <h5>Reference</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/release/">Changelog and Release Notes</a> |
| </li> |
| <li class="none"> |
| <a href="../../../api/index.html">Java API docs</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/UnixShellAPI.html">Unix Shell API</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/Metrics.html">Metrics</a> |
| </li> |
| </ul> |
| <h5>Configuration</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/core-default.xml">core-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/hdfs-default.xml">hdfs-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs-rbf/hdfs-rbf-default.xml">hdfs-rbf-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml">mapred-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-common/yarn-default.xml">yarn-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-kms/kms-default.html">kms-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-hdfs-httpfs/httpfs-default.html">httpfs-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/DeprecatedProperties.html">Deprecated Properties</a> |
| </li> |
| </ul> |
| <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"> |
| <img alt="Built by Maven" src="../images/logos/maven-feather.png"/> |
| </a> |
| |
| </div> |
| </div> |
| <div id="bodyColumn"> |
| <div id="contentBox"> |
| <!--- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. See accompanying LICENSE file. |
| --> |
| <h1>Statistic collection with the IOStatistics API</h1> |
| |
| <div> |
| <div> |
| <pre class="source">@InterfaceAudience.Public |
| @InterfaceStability.Unstable |
| </pre></div></div> |
| |
| <p>The <tt>IOStatistics</tt> API is intended to provide statistics on individual IO classes -such as input and output streams, <i>in a standard way which applications can query</i></p> |
| <p>Many filesystem-related classes have implemented statistics gathering and provided private/unstable ways to query this, but as they were not common across implementations it was unsafe for applications to reference these values. Example: <tt>S3AInputStream</tt> and its statistics API. This is used in internal tests, but cannot be used downstream in applications such as Apache Hive or Apache HBase.</p> |
| <p>The IOStatistics API is intended to</p> |
| <ol style="list-style-type: decimal"> |
| |
| <li>Be instance specific:, rather than shared across multiple instances of a class, or thread local.</li> |
| <li>Be public and stable enough to be used by applications.</li> |
| <li>Be easy to use in applications written in Java, Scala, and, via libhdfs, C/C++</li> |
| <li>Have foundational interfaces and classes in the <tt>hadoop-common</tt> JAR.</li> |
| </ol> |
| <div class="section"> |
| <h2><a name="Core_Model"></a>Core Model</h2> |
| <p>Any class <i>may</i> implement <tt>IOStatisticsSource</tt> in order to provide statistics.</p> |
| <p>Wrapper I/O Classes such as <tt>FSDataInputStream</tt> anc <tt>FSDataOutputStream</tt> <i>should</i> implement the interface and forward it to the wrapped class, if they also implement it -and return <tt>null</tt> if they do not.</p> |
| <p><tt>IOStatisticsSource</tt> implementations <tt>getIOStatistics()</tt> return an instance of <tt>IOStatistics</tt> enumerating the statistics of that specific instance.</p> |
| <p>The <tt>IOStatistics</tt> Interface exports five kinds of statistic:</p> |
| <table border="0" class="bodyTable"> |
| <thead> |
| |
| <tr class="a"> |
| <th> Category </th> |
| <th> Type </th> |
| <th> Description </th></tr> |
| </thead><tbody> |
| |
| <tr class="b"> |
| <td> <tt>counter</tt> </td> |
| <td> <tt>long</tt> </td> |
| <td> a counter which may increase in value; SHOULD BE >= 0 </td></tr> |
| <tr class="a"> |
| <td> <tt>gauge</tt> </td> |
| <td> <tt>long</tt> </td> |
| <td> an arbitrary value which can down as well as up; SHOULD BE >= 0 </td></tr> |
| <tr class="b"> |
| <td> <tt>minimum</tt> </td> |
| <td> <tt>long</tt> </td> |
| <td> an minimum value; MAY BE negative </td></tr> |
| <tr class="a"> |
| <td> <tt>maximum</tt> </td> |
| <td> <tt>long</tt> </td> |
| <td> a maximum value; MAY BE negative </td></tr> |
| <tr class="b"> |
| <td> <tt>meanStatistic</tt> </td> |
| <td> <tt>MeanStatistic</tt> </td> |
| <td> an arithmetic mean and sample size; mean MAY BE negative </td></tr> |
| </tbody> |
| </table> |
| <p>Four are simple <tt>long</tt> values, with the variations how they are likely to change and how they are aggregated.</p> |
| <div class="section"> |
| <div class="section"> |
| <h4><a name="Aggregation_of_Statistic_Values"></a>Aggregation of Statistic Values</h4> |
| <p>For the different statistic category, the result of <tt>aggregate(x, y)</tt> is</p> |
| <table border="0" class="bodyTable"> |
| <thead> |
| |
| <tr class="a"> |
| <th> Category </th> |
| <th> Aggregation </th></tr> |
| </thead><tbody> |
| |
| <tr class="b"> |
| <td> <tt>counter</tt> </td> |
| <td> <tt>max(0, x) + max(0, y)</tt> </td></tr> |
| <tr class="a"> |
| <td> <tt>gauge</tt> </td> |
| <td> <tt>max(0, x) + max(0, y)</tt> </td></tr> |
| <tr class="b"> |
| <td> <tt>minimum</tt> </td> |
| <td> <tt>min(x, y)</tt> </td></tr> |
| <tr class="a"> |
| <td> <tt>maximum</tt> </td> |
| <td> <tt>max(x, y)</tt> </td></tr> |
| <tr class="b"> |
| <td> <tt>meanStatistic</tt> </td> |
| <td> calculation of the mean of <tt>x</tt> and <tt>y</tt> ) </td></tr> |
| </tbody> |
| </table></div> |
| <div class="section"> |
| <h4><a name="Class_MeanStatistic"></a>Class <tt>MeanStatistic</tt></h4></div></div></div> |
| <div class="section"> |
| <h2><a name="package_org.apache.hadoop.fs.statistics"></a>package <tt>org.apache.hadoop.fs.statistics</tt></h2> |
| <p>This package contains the public statistics APIs intended for use by applications.</p><!-- ============================================================= --> |
| <!-- Class: MeanStatistic --> |
| <!-- ============================================================= --> |
| |
| <p><tt>MeanStatistic</tt> is a tuple of <tt>(mean, samples)</tt> to support aggregation.</p> |
| <p>A <tt>MeanStatistic</tt> with a sample of <tt>0</tt> is considered an empty statistic.</p> |
| <p>All <tt>MeanStatistic</tt> instances where <tt>sample = 0</tt> are considered equal, irrespective of the <tt>mean</tt> value.</p> |
| <p>Algorithm to calculate the mean :</p> |
| |
| <div> |
| <div> |
| <pre class="source">if x.samples = 0: |
| y |
| else if y.samples = 0 : |
| x |
| else: |
| samples' = x.samples + y.samples |
| mean' = (x.mean * x.samples) + (y.mean * y.samples) / samples' |
| (samples', mean') |
| </pre></div></div> |
| |
| <p>Implicitly, this means that if both samples are empty, then the aggregate value is also empty.</p> |
| |
| <div> |
| <div> |
| <pre class="source">public final class MeanStatistic implements Serializable, Cloneable { |
| /** |
| * Arithmetic mean. |
| */ |
| private double mean; |
| |
| /** |
| * Number of samples used to calculate |
| * the mean. |
| */ |
| private long samples; |
| |
| /** |
| * Get the mean value. |
| * @return the mean |
| */ |
| public double getMean() { |
| return mean; |
| } |
| |
| /** |
| * Get the sample count. |
| * @return the sample count; 0 means empty |
| */ |
| public long getSamples() { |
| return samples; |
| } |
| |
| /** |
| * Is a statistic empty? |
| * @return true if the sample count is 0 |
| */ |
| public boolean isEmpty() { |
| return samples == 0; |
| } |
| /** |
| * Add another mean statistic to create a new statistic. |
| * When adding two statistics, if either is empty then |
| * a copy of the non-empty statistic is returned. |
| * If both are empty then a new empty statistic is returned. |
| * |
| * @param other other value |
| * @return the aggregate mean |
| */ |
| public MeanStatistic add(final MeanStatistic other) { |
| /* Implementation elided. */ |
| } |
| @Override |
| public int hashCode() { |
| return Objects.hash(mean, samples); |
| } |
| |
| @Override |
| public boolean equals(final Object o) { |
| if (this == o) { return true; } |
| if (o == null || getClass() != o.getClass()) { return false; } |
| MeanStatistic that = (MeanStatistic) o; |
| if (this.isEmpty()) { |
| return that.isEmpty(); |
| } |
| return Double.compare(that.mean, mean) == 0 && |
| samples == that.samples; |
| } |
| |
| @Override |
| public MeanStatistic clone() { |
| return new MeanStatistic(this); |
| } |
| |
| public MeanStatistic copy() { |
| return new MeanStatistic(this); |
| } |
| |
| } |
| </pre></div></div> |
| <!-- ============================================================= --> |
| <!-- Interface: IOStatisticsSource --> |
| <!-- ============================================================= --> |
| |
| <div class="section"> |
| <h3><a name="class_org.apache.hadoop.fs.statistics.IOStatisticsSource"></a>class <tt>org.apache.hadoop.fs.statistics.IOStatisticsSource</tt></h3> |
| |
| <div> |
| <div> |
| <pre class="source">/** |
| * A source of IO statistics. |
| * These statistics MUST be instance specific, not thread local. |
| */ |
| @InterfaceStability.Unstable |
| public interface IOStatisticsSource { |
| |
| /** |
| * Return a statistics instance. |
| * It is not a requirement that the same instance is returned every time. |
| * {@link IOStatisticsSource}. |
| * If the object implementing this is Closeable, this method |
| * may return null if invoked on a closed object, even if |
| * it returns a valid instance when called earlier. |
| * @return an IOStatistics instance or null |
| */ |
| IOStatistics getIOStatistics(); |
| } |
| </pre></div></div> |
| |
| <p>This is the interface which an object instance MUST implement if they are a source of IOStatistics information.</p> |
| <div class="section"> |
| <h4><a name="Invariants"></a>Invariants</h4> |
| <p>The result of <tt>getIOStatistics()</tt> must be one of</p> |
| <ul> |
| |
| <li><tt>null</tt></li> |
| <li>an immutable <tt>IOStatistics</tt> for which each map of entries is an empty map.</li> |
| <li>an instance of an <tt>IOStatistics</tt> whose statistics MUST BE unique to that instance of the class implementing <tt>IOStatisticsSource</tt>.</li> |
| </ul> |
| <p>Less formally: if the statistics maps returned are non-empty, all the statistics must be collected from the current instance, and not from any other instances, the way some of the <tt>FileSystem</tt> statistics are collected.</p> |
| <p>The result of <tt>getIOStatistics()</tt>, if non-null, MAY be a different instance on every invocation.</p><!-- ============================================================= --> |
| <!-- Interface: IOStatistics --> |
| <!-- ============================================================= --> |
| </div></div> |
| <div class="section"> |
| <h3><a name="class_org.apache.hadoop.fs.statistics.IOStatistics"></a>class <tt>org.apache.hadoop.fs.statistics.IOStatistics</tt></h3> |
| <p>These are per-instance statistics provided by an object which implements <tt>IOStatisticsSource</tt>.</p> |
| |
| <div> |
| <div> |
| <pre class="source">@InterfaceAudience.Public |
| @InterfaceStability.Unstable |
| public interface IOStatistics { |
| |
| /** |
| * Map of counters. |
| * @return the current map of counters. |
| */ |
| Map<String, Long> counters(); |
| |
| /** |
| * Map of gauges. |
| * @return the current map of gauges. |
| */ |
| Map<String, Long> gauges(); |
| |
| /** |
| * Map of minumums. |
| * @return the current map of minumums. |
| */ |
| Map<String, Long> minumums(); |
| |
| /** |
| * Map of maximums. |
| * @return the current map of maximums. |
| */ |
| Map<String, Long> maximums(); |
| |
| /** |
| * Map of meanStatistics. |
| * @return the current map of MeanStatistic statistics. |
| */ |
| Map<String, MeanStatistic> meanStatistics(); |
| |
| } |
| </pre></div></div> |
| </div> |
| <div class="section"> |
| <h3><a name="Statistic_Naming"></a>Statistic Naming</h3> |
| <p>The naming policy of statistics is designed to be readable, shareable and ideally consistent across <tt>IOStatisticSource</tt> implementations.</p> |
| <ul> |
| |
| <li> |
| |
| <p>Characters in key names MUST match the regular expression <tt>[a-z|0-9|_]</tt> with the exception of the first character, which MUST be in the range <tt>[a-z]</tt>. Thus the full regular expression for a valid statistic name is:</p> |
| |
| <div> |
| <div> |
| <pre class="source">[a-z][a-z|0-9|_]+ |
| </pre></div></div> |
| </li> |
| <li> |
| |
| <p>Where possible, the names of statistics SHOULD be those defined with common names.</p> |
| |
| <div> |
| <div> |
| <pre class="source">org.apache.hadoop.fs.statistics.StreamStatisticNames |
| org.apache.hadoop.fs.statistics.StoreStatisticNames |
| </pre></div></div> |
| </li> |
| </ul> |
| <p>Note 1.: these are evolving; for clients to safely reference their statistics by name they SHOULD be copied to the application. (i.e. for an application compiled hadoop 3.4.2 to link against hadoop 3.4.1, copy the strings).</p> |
| <p>Note 2: keys defined in these classes SHALL NOT be removed from subsequent Hadoop releases.</p> |
| <ul> |
| |
| <li> |
| |
| <p>A common statistic name MUST NOT be used to report any other statistic and MUST use the pre-defined unit of measurement.</p> |
| </li> |
| <li> |
| |
| <p>A statistic name in one of the maps SHOULD NOT be re-used in another map. This aids diagnostics of logged statistics.</p> |
| </li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="Statistic_Maps"></a>Statistic Maps</h3> |
| <p>For each map of statistics returned:</p> |
| <ul> |
| |
| <li> |
| |
| <p>The operations to add/remove entries are unsupported: the map returned MAY be mutable by the source of statistics.</p> |
| </li> |
| <li> |
| |
| <p>The map MAY be empty.</p> |
| </li> |
| <li> |
| |
| <p>The map keys each represent a measured statistic.</p> |
| </li> |
| <li> |
| |
| <p>The set of keys in a map SHOULD remain unchanged, and MUST NOT remove keys.</p> |
| </li> |
| <li> |
| |
| <p>The statistics SHOULD be dynamic: every lookup of an entry SHOULD return the latest value.</p> |
| </li> |
| <li> |
| |
| <p>The values MAY change across invocations of <tt>Map.values()</tt> and <tt>Map.entries()</tt></p> |
| </li> |
| <li> |
| |
| <p>The update MAY be in the <tt>iterable()</tt> calls of the iterators returned, or MAY be in the actual <tt>iterable.next()</tt> operation. That is: there is no guarantee as to when the evaluation takes place.</p> |
| </li> |
| <li> |
| |
| <p>The returned <tt>Map.Entry</tt> instances MUST return the same value on repeated <tt>getValue()</tt> calls. (i.e once you have the entry, it is immutable).</p> |
| </li> |
| <li> |
| |
| <p>Queries of statistics SHOULD be fast and non-blocking to the extent that if invoked during a long operation, they will prioritize returning fast over most timely values.</p> |
| </li> |
| <li> |
| |
| <p>The statistics MAY lag; especially for statistics collected in separate operations (e.g stream IO statistics as provided by a filesystem instance).</p> |
| </li> |
| <li> |
| |
| <p>Statistics which represent time SHOULD use milliseconds as their unit.</p> |
| </li> |
| <li> |
| |
| <p>Statistics which represent time and use a different unit MUST document the unit used.</p> |
| </li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="Thread_Model"></a>Thread Model</h3> |
| <ol style="list-style-type: decimal"> |
| |
| <li> |
| |
| <p>An instance of <tt>IOStatistics</tt> can be shared across threads;</p> |
| </li> |
| <li> |
| |
| <p>Read access to the supplied statistics maps MUST be thread safe.</p> |
| </li> |
| <li> |
| |
| <p>Iterators returned from the maps MUST NOT be shared across threads.</p> |
| </li> |
| <li> |
| |
| <p>The statistics collected MUST include all operations which took place across all threads performing work for the monitored object.</p> |
| </li> |
| <li> |
| |
| <p>The statistics reported MUST NOT be local to the active thread.</p> |
| </li> |
| </ol> |
| <p>This is different from the <tt>FileSystem.Statistics</tt> behavior where per-thread statistics are collected and reported.</p> |
| <p>That mechanism supports collecting limited read/write statistics for different worker threads sharing the same FS instance, but as the collection is thread local, it invariably under-reports IO performed in other threads on behalf of a worker thread.</p></div></div> |
| <div class="section"> |
| <h2><a name="Statisic_Snapshot"></a>Statisic Snapshot</h2> |
| <p>A snapshot of the current statistic values MAY be obtained by calling <tt>IOStatisticsSupport.snapshotIOStatistics()</tt></p> |
| |
| <div> |
| <div> |
| <pre class="source"> public static <X extends IOStatistics & Serializable> X |
| snapshotIOStatistics(IOStatistics statistics) |
| </pre></div></div> |
| |
| <p>This snapshot is serializable through Java serialization and through Jackson to/from JSON.</p></div> |
| <div class="section"> |
| <h2><a name="Helper_Classes"></a>Helper Classes</h2> |
| <div class="section"> |
| <h3><a name="class_org.apache.hadoop.fs.statistics.IOStatisticsSupport"></a>class <tt>org.apache.hadoop.fs.statistics.IOStatisticsSupport</tt></h3> |
| <p>This provides helper methods to work with IOStatistics sources and instances.</p> |
| <p>Consult the javadocs for its operations.</p></div> |
| <div class="section"> |
| <h3><a name="class_org.apache.hadoop.fs.statistics.IOStatisticsLogging"></a>class <tt>org.apache.hadoop.fs.statistics.IOStatisticsLogging</tt></h3> |
| <p>Support for efficiently logging <tt>IOStatistics</tt>/<tt>IOStatisticsSource</tt> instances.</p> |
| <p>These are intended for assisting logging, including only enumerating the state of an <tt>IOStatistics</tt> instance when the log level needs it.</p> |
| |
| <div> |
| <div> |
| <pre class="source">LOG.info("IOStatistics after upload: {}", demandStringify(iostats)); |
| |
| // or even better, as it results in only a single object creations |
| Object latest = demandStringify(iostats); |
| LOG.info("IOStatistics : {}", latest); |
| /* do some work. */ |
| LOG.info("IOStatistics : {}", latest); |
| |
| </pre></div></div> |
| </div></div> |
| <div class="section"> |
| <h2><a name="Package_org.apache.hadoop.fs.statistics.impl"></a>Package <tt>org.apache.hadoop.fs.statistics.impl</tt></h2> |
| <p>This contains implementation classes to support providing statistics to applications.</p> |
| <p>These MUST NOT BE used by applications. If a feature is needed from this package then the provisioning of a public implementation MAY BE raised via the Hadoop development channels.</p> |
| <p>These MAY be used by those implementations of the Hadoop <tt>FileSystem</tt>, <tt>AbstractFileSystem</tt> and related classes which are not in the hadoop source tree. Implementors MUST BE aware that the implementation this code is unstable and may change across minor point releases of Hadoop.</p></div> |
| </div> |
| </div> |
| <div class="clear"> |
| <hr/> |
| </div> |
| <div id="footer"> |
| <div class="xright"> |
| © 2008-2021 |
| Apache Software Foundation |
| |
| - <a href="http://maven.apache.org/privacy-policy.html">Privacy Policy</a>. |
| Apache Maven, Maven, Apache, the Apache feather logo, and the Apache Maven project logos are trademarks of The Apache Software Foundation. |
| </div> |
| <div class="clear"> |
| <hr/> |
| </div> |
| </div> |
| </body> |
| </html> |
