| <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
| <!-- |
| | Generated by Apache Maven Doxia at 2021-06-15 |
| | Rendered using Apache Maven Stylus Skin 1.5 |
| --> |
| <html xmlns="http://www.w3.org/1999/xhtml"> |
| <head> |
| <title>Apache Hadoop 3.3.1 – View File System Overload Scheme Guide</title> |
| <style type="text/css" media="all"> |
| @import url("./css/maven-base.css"); |
| @import url("./css/maven-theme.css"); |
| @import url("./css/site.css"); |
| </style> |
| <link rel="stylesheet" href="./css/print.css" type="text/css" media="print" /> |
| <meta name="Date-Revision-yyyymmdd" content="20210615" /> |
| <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> |
| </head> |
| <body class="composite"> |
| <div id="banner"> |
| <a href="http://hadoop.apache.org/" id="bannerLeft"> |
| <img src="http://hadoop.apache.org/images/hadoop-logo.jpg" alt="" /> |
| </a> |
| <a href="http://www.apache.org/" id="bannerRight"> |
| <img src="http://www.apache.org/images/asf_logo_wide.png" alt="" /> |
| </a> |
| <div class="clear"> |
| <hr/> |
| </div> |
| </div> |
| <div id="breadcrumbs"> |
| |
| <div class="xleft"> |
| <a href="http://www.apache.org/" class="externalLink">Apache</a> |
| > |
| <a href="http://hadoop.apache.org/" class="externalLink">Hadoop</a> |
| > |
| <a href="../index.html">Apache Hadoop Project Dist POM</a> |
| > |
| <a href="index.html">Apache Hadoop 3.3.1</a> |
| > |
| View File System Overload Scheme Guide |
| </div> |
| <div class="xright"> <a href="http://wiki.apache.org/hadoop" class="externalLink">Wiki</a> |
| | |
| <a href="https://gitbox.apache.org/repos/asf/hadoop.git" class="externalLink">git</a> |
| | |
| <a href="http://hadoop.apache.org/" class="externalLink">Apache Hadoop</a> |
| |
| | Last Published: 2021-06-15 |
| | Version: 3.3.1 |
| </div> |
| <div class="clear"> |
| <hr/> |
| </div> |
| </div> |
| <div id="leftColumn"> |
| <div id="navcolumn"> |
| |
| <h5>General</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../index.html">Overview</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/SingleCluster.html">Single Node Setup</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/ClusterSetup.html">Cluster Setup</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/CommandsManual.html">Commands Reference</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/FileSystemShell.html">FileSystem Shell</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/Compatibility.html">Compatibility Specification</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/DownstreamDev.html">Downstream Developer's Guide</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/AdminCompatibilityGuide.html">Admin Compatibility Guide</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/InterfaceClassification.html">Interface Classification</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/filesystem/index.html">FileSystem Specification</a> |
| </li> |
| </ul> |
| <h5>Common</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/CLIMiniCluster.html">CLI Mini Cluster</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/FairCallQueue.html">Fair Call Queue</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/NativeLibraries.html">Native Libraries</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/Superusers.html">Proxy User</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/RackAwareness.html">Rack Awareness</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/SecureMode.html">Secure Mode</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/ServiceLevelAuth.html">Service Level Authorization</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/HttpAuthentication.html">HTTP Authentication</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/CredentialProviderAPI.html">Credential Provider API</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-kms/index.html">Hadoop KMS</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/Tracing.html">Tracing</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/UnixShellGuide.html">Unix Shell Guide</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/registry/index.html">Registry</a> |
| </li> |
| </ul> |
| <h5>HDFS</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsDesign.html">Architecture</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html">User Guide</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HDFSCommands.html">Commands Reference</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html">NameNode HA With QJM</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithNFS.html">NameNode HA With NFS</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/ObserverNameNode.html">Observer NameNode</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/Federation.html">Federation</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/ViewFs.html">ViewFs</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/ViewFsOverloadScheme.html">ViewFsOverloadScheme</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsSnapshots.html">Snapshots</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsEditsViewer.html">Edits Viewer</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsImageViewer.html">Image Viewer</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html">Permissions and HDFS</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html">Quotas and HDFS</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/LibHdfs.html">libhdfs (C API)</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/WebHDFS.html">WebHDFS (REST API)</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-hdfs-httpfs/index.html">HttpFS</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html">Short Circuit Local Reads</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/CentralizedCacheManagement.html">Centralized Cache Management</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html">NFS Gateway</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html">Rolling Upgrade</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/ExtendedAttributes.html">Extended Attributes</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html">Transparent Encryption</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsMultihoming.html">Multihoming</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html">Storage Policies</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/MemoryStorage.html">Memory Storage Support</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/SLGUserGuide.html">Synthetic Load Generator</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html">Erasure Coding</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HDFSDiskbalancer.html">Disk Balancer</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsUpgradeDomain.html">Upgrade Domain</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsDataNodeAdminGuide.html">DataNode Admin</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs-rbf/HDFSRouterFederation.html">Router Federation</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsProvidedStorage.html">Provided Storage</a> |
| </li> |
| </ul> |
| <h5>MapReduce</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html">Tutorial</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredCommands.html">Commands Reference</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html">Compatibility with 1.x</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html">Encrypted Shuffle</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html">Pluggable Shuffle/Sort</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/DistributedCacheDeploy.html">Distributed Cache Deploy</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/SharedCacheSupport.html">Support for YARN Shared Cache</a> |
| </li> |
| </ul> |
| <h5>MapReduce REST APIs</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredAppMasterRest.html">MR Application Master</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-hs/HistoryServerRest.html">MR History Server</a> |
| </li> |
| </ul> |
| <h5>YARN</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/YARN.html">Architecture</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/YarnCommands.html">Commands Reference</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html">Capacity Scheduler</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/FairScheduler.html">Fair Scheduler</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html">ResourceManager Restart</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html">ResourceManager HA</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/ResourceModel.html">Resource Model</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/NodeLabel.html">Node Labels</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/NodeAttributes.html">Node Attributes</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/WebApplicationProxy.html">Web Application Proxy</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/TimelineServer.html">Timeline Server</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html">Timeline Service V.2</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html">Writing YARN Applications</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html">YARN Application Security</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/NodeManager.html">NodeManager</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/DockerContainers.html">Running Applications in Docker Containers</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/RuncContainers.html">Running Applications in runC Containers</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/NodeManagerCgroups.html">Using CGroups</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/SecureContainer.html">Secure Containers</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/ReservationSystem.html">Reservation System</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/GracefulDecommission.html">Graceful Decommission</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html">Opportunistic Containers</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/Federation.html">YARN Federation</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/SharedCache.html">Shared Cache</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/UsingGpus.html">Using GPU</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/UsingFPGA.html">Using FPGA</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html">Placement Constraints</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/YarnUI2.html">YARN UI2</a> |
| </li> |
| </ul> |
| <h5>YARN REST APIs</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html">Introduction</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html">Resource Manager</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/NodeManagerRest.html">Node Manager</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/TimelineServer.html#Timeline_Server_REST_API_v1">Timeline Server</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html#Timeline_Service_v.2_REST_API">Timeline Service V.2</a> |
| </li> |
| </ul> |
| <h5>YARN Service</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/Overview.html">Overview</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/QuickStart.html">QuickStart</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/Concepts.html">Concepts</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/YarnServiceAPI.html">Yarn Service API</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/ServiceDiscovery.html">Service Discovery</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/SystemServices.html">System Services</a> |
| </li> |
| </ul> |
| <h5>Hadoop Compatible File Systems</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-aliyun/tools/hadoop-aliyun/index.html">Aliyun OSS</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-aws/tools/hadoop-aws/index.html">Amazon S3</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-azure/index.html">Azure Blob Storage</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-azure-datalake/index.html">Azure Data Lake Storage</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-openstack/index.html">OpenStack Swift</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-cos/cloud-storage/index.html">Tencent COS</a> |
| </li> |
| </ul> |
| <h5>Auth</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-auth/index.html">Overview</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-auth/Examples.html">Examples</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-auth/Configuration.html">Configuration</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-auth/BuildingIt.html">Building</a> |
| </li> |
| </ul> |
| <h5>Tools</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-streaming/HadoopStreaming.html">Hadoop Streaming</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-archives/HadoopArchives.html">Hadoop Archives</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-archive-logs/HadoopArchiveLogs.html">Hadoop Archive Logs</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-distcp/DistCp.html">DistCp</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-gridmix/GridMix.html">GridMix</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-rumen/Rumen.html">Rumen</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-resourceestimator/ResourceEstimator.html">Resource Estimator Service</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-sls/SchedulerLoadSimulator.html">Scheduler Load Simulator</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/Benchmarking.html">Hadoop Benchmarking</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-dynamometer/Dynamometer.html">Dynamometer</a> |
| </li> |
| </ul> |
| <h5>Reference</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/release/">Changelog and Release Notes</a> |
| </li> |
| <li class="none"> |
| <a href="../../api/index.html">Java API docs</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/UnixShellAPI.html">Unix Shell API</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/Metrics.html">Metrics</a> |
| </li> |
| </ul> |
| <h5>Configuration</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/core-default.xml">core-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/hdfs-default.xml">hdfs-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs-rbf/hdfs-rbf-default.xml">hdfs-rbf-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml">mapred-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-common/yarn-default.xml">yarn-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-kms/kms-default.html">kms-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-hdfs-httpfs/httpfs-default.html">httpfs-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/DeprecatedProperties.html">Deprecated Properties</a> |
| </li> |
| </ul> |
| <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"> |
| <img alt="Built by Maven" src="./images/logos/maven-feather.png"/> |
| </a> |
| |
| </div> |
| </div> |
| <div id="bodyColumn"> |
| <div id="contentBox"> |
| <!--- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. See accompanying LICENSE file. |
| --> |
| <h1>View File System Overload Scheme Guide</h1> |
| <ul> |
| <li><a href="#Introduction">Introduction</a></li> |
| <li><a href="#View_File_System_Overload_Scheme">View File System Overload Scheme</a> |
| <ul> |
| <li><a href="#Details">Details</a></li> |
| <li><a href="#Enabling_View_File_System_Overload_Scheme">Enabling View File System Overload Scheme</a></li> |
| <li><a href="#Example_Configurations">Example Configurations</a></li> |
| <li><a href="#Central_Mount_Table_Configurations">Central Mount Table Configurations</a></li></ul></li> |
| <li><a href="#DFSAdmin_commands_with_View_File_System_Overload_Scheme">DFSAdmin commands with View File System Overload Scheme</a></li> |
| <li><a href="#Accessing_paths_without_authority">Accessing paths without authority</a> |
| <ul> |
| <li><a href="#Solution">Solution</a></li></ul></li> |
| <li><a href="#Appendix:_A_Mount_Table_Configuration_with_XInclude">Appendix: A Mount Table Configuration with XInclude</a></li></ul> |
| |
| <div class="section"> |
| <h2><a name="Introduction"></a>Introduction</h2> |
| <p>The View File System Overload Scheme introduced to solve two key challenges with the View File System(ViewFS). The first problem is, to use ViewFS, users need to update fs.defaultFS with viewfs scheme (<tt>viewfs://</tt>). The second problem is that users need to copy the mount-table configurations to all the client nodes. The ViewFileSystemOverloadScheme is addressing these challenges.</p></div> |
| <div class="section"> |
| <h2><a name="View_File_System_Overload_Scheme"></a>View File System Overload Scheme</h2> |
| <div class="section"> |
| <h3><a name="Details"></a>Details</h3> |
| <p>The View File System Overload Scheme is an extension to the View File System. This will allow users to continue to use their existing fs.defaultFS configured scheme or any new scheme name instead of using scheme <tt>viewfs</tt>. Mount link configurations key, value formats are same as in <a href="./ViewFs.html">ViewFS Guide</a>. If a user wants to continue use the same fs.defaultFS and wants to have more mount points, then mount link configurations should have the ViewFileSystemOverloadScheme initialized uri’s hostname as the mount table name. Example if fs.defaultFS is <tt>hdfs://mycluster</tt>, then the mount link configuration key name should be like in the following format <tt>fs.viewfs.mounttable.*mycluster*.link.<mountLinkPath></tt>. Even if the initialized fs uri has hostname:port, it will simply ignore the port number and only consider the hostname as the mount table name. We will discuss more example configurations in following sections. If there are no mount links configured with the initializing uri’s hostname as the mount table name, then it will automatically consider the current uri as fallback(<tt>fs.viewfs.mounttable.*mycluster*.linkFallback</tt>) target fs uri. If the initialized uri contains path part, it will consider only scheme and authority part, but not the path part. Example, if the initialized uri contains <tt>hdfs://mycluster/data</tt>, it will consider only <tt>hdfs://mycluster</tt> as fallback target fs uri. The path part <tt>data</tt> will be ignored.</p> |
| <p>Another important improvement with the ViewFileSystemOverloadScheme is, administrators need not copy the <tt>mount-table.xml</tt> configuration file to 1000s of client nodes. Instead, they can keep the mount-table configuration file in a Hadoop compatible file system. So, keeping the configuration file in a central place makes administrators life easier as they can update mount-table in single place.</p></div> |
| <div class="section"> |
| <h3><a name="Enabling_View_File_System_Overload_Scheme"></a>Enabling View File System Overload Scheme</h3> |
| <p>To use this class, the following configurations needed to be added in core-site.xml file.</p> |
| |
| <div> |
| <div> |
| <pre class="source"><property> |
| <name>fs.<scheme>.impl</name> |
| <value>org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme</value> |
| </property> |
| </pre></div></div> |
| |
| <p>Here <tt><scheme></tt> should be same as the uri-scheme configured in fs.defautFS. For example if fs.defaultFS was configured with <tt>hdfs://mycluster</tt>, then the above configuration would be like below:</p> |
| |
| <div> |
| <div> |
| <pre class="source"><property> |
| <name>fs.hdfs.impl</name> |
| <value>org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme</value> |
| </property> |
| </pre></div></div> |
| </div> |
| <div class="section"> |
| <h3><a name="Example_Configurations"></a>Example Configurations</h3> |
| <p><b>Example 1:</b></p> |
| <p>If users want some of their existing cluster (<tt>hdfs://cluster</tt>) data to mount with hdfs(<tt>hdfs://cluster</tt>) and other object store clusters(<tt>o3fs://bucket1.volume1.omhost/</tt>, <tt>s3a://bucket1/</tt>), the following example configurations can show how to add mount links.</p> |
| |
| <div> |
| <div> |
| <pre class="source"><property> |
| <name>fs.viewfs.mounttable.cluster.link./user</name> |
| <value>hdfs://cluster/user</value> |
| </property> |
| |
| <property> |
| <name>fs.viewfs.mounttable.cluster.link./data</name> |
| <value>o3fs://bucket1.volume1/data</value> |
| </property> |
| |
| <property> |
| <name>fs.viewfs.mounttable.cluster.link./backup</name> |
| <value>s3a://bucket1/backup/</value> |
| </property> |
| </pre></div></div> |
| |
| <p>Let’s consider the following operations to understand where these operations will be delegated based on mount links.</p> |
| <p><i>Op1:</i> Create a file with the the path <tt>hdfs://cluster/user/fileA</tt>, then physically this file will be created at <tt>hdfs://cluster/user/fileA</tt>. This delegation happened based on the first configuration parameter in above configurations. Here <tt>/user</tt> mapped to <tt>hdfs://cluster/user/</tt>.</p> |
| <p><i>Op2:</i> Create a file the the path <tt>hdfs://cluster/data/datafile</tt>, then this file will be created at <tt>o3fs://bucket1.volume1.omhost/data/datafile</tt>. This delegation happened based on second configurations parameter in above configurations. Here <tt>/data</tt> was mapped with <tt>o3fs://bucket1.volume1.omhost/data/</tt>.</p> |
| <p><i>Op3:</i> Create a file with the the path <tt>hdfs://cluster/backup/data.zip</tt>, then physically this file will be created at <tt>s3a://bucket1/backup/data.zip</tt>. This delegation happened based on the third configuration parameter in above configurations. Here <tt>/backup</tt> was mapped to <tt>s3a://bucket1/backup/</tt>.</p> |
| <p><b>Example 2:</b></p> |
| <p>If users want some of their existing cluster (<tt>s3a://bucketA/</tt>) data to mount with other hdfs cluster(<tt>hdfs://cluster</tt>) and object store clusters(<tt>o3fs://bucket1.volume1.omhost/</tt>, <tt>s3a://bucketA/</tt>), the following example configurations can show how to add mount links.</p> |
| |
| <div> |
| <div> |
| <pre class="source"><property> |
| <name>fs.viewfs.mounttable.bucketA.link./user</name> |
| <value>hdfs://cluster/user</value> |
| </property> |
| |
| <property> |
| <name>fs.viewfs.mounttable.bucketA.link./data</name> |
| <value>o3fs://bucket1.volume1.omhost/data</value> |
| </property> |
| |
| <property> |
| <name>fs.viewfs.mounttable.bucketA.link./salesDB</name> |
| <value>s3a://bucketA/salesDB/</value> |
| </property> |
| </pre></div></div> |
| |
| <p>Let’s consider the following operations to understand to where these operations will be delegated based on mount links.</p> |
| <p><i>Op1:</i> Create a file with the the path <tt>s3a://bucketA/user/fileA</tt>, then this file will be created physically at <tt>hdfs://cluster/user/fileA</tt>. This delegation happened based on the first configuration parameter in above configurations. Here <tt>/user</tt> mapped to <tt>hdfs://cluster/user</tt>.</p> |
| <p><i>Op2:</i> Create a file the the path <tt>s3a://bucketA/data/datafile</tt>, then this file will be created at <tt>o3fs://bucket1.volume1.omhost/data/datafile</tt>. This delegation happened based on second configurations parameter in above configurations. Here <tt>/data</tt> was mapped with <tt>o3fs://bucket1.volume1.omhost/data/</tt>.</p> |
| <p><i>Op3:</i> Create a file with the the path <tt>s3a://bucketA/salesDB/dbfile</tt>, then physically this file will be created at <tt>s3a://bucketA/salesDB/dbfile</tt>. This delegation happened based on the third configuration parameter in above configurations. Here <tt>/salesDB</tt> was mapped to <tt>s3a://bucket1/salesDB</tt>.</p> |
| <p>Note: In above examples we used create operation only, but the same mechanism applies to any other file system APIs here.</p> |
| <p>The following picture shows how the different schemes can be used in ViewFileSystemOverloadScheme compared to the ViewFileSystem.</p> |
| <img src="./images/ViewFSOverloadScheme.png" width="1050" height="550" alt="" /> |
| |
| <p>Note: In ViewFsOverloadScheme, by default the mount links will not be represented as symlinks. The permission bits and isDirectory value will be propagated from the target directory/file.</p></div> |
| <div class="section"> |
| <h3><a name="Central_Mount_Table_Configurations"></a>Central Mount Table Configurations</h3> |
| <p>To enable central mount table configuration, we need to configure <tt>fs.viewfs.mounttable.path</tt> in <tt>core-site.xml</tt> with the value as the Hadoop compatible file system directory/file path, where the <tt>mount-table.<versionNumber>.xml</tt> file copied. Here versionNumber is an integer number and need to increase the version number and upload new file in same directory.</p> |
| <p>The ViewFileSystemOverloadScheme always loads the highest version number <tt>mount-table.<versionNumber>.xml</tt>. Please don’t replace the file with same name. Always increment the version number to take new file picked by newly initializing clients. Why we don’t recommend to replace the files is that, some client might have already opened the connections to old mount-table files already and in middle of loading configuration files, and replacing files can make them fail.</p> |
| |
| <div> |
| <div> |
| <pre class="source"><property> |
| <name>fs.viewfs.mounttable.path</name> |
| <value>hdfs://cluster/config/mount-table-dir</value> |
| </property> |
| </pre></div></div> |
| |
| <p>If you are sure, you will never do updates to mount-table file, you can also configure file path directly like below. If you configure file path, it will not check any highest version number loading. Whatever file configured it will be loaded. However file name format should be same.</p> |
| |
| <div> |
| <div> |
| <pre class="source"><property> |
| <name>fs.viewfs.mounttable.path</name> |
| <value>hdfs://cluster/config/mount-table-dir/mount-table.<versionNumber>.xml</value> |
| </property> |
| </pre></div></div> |
| |
| <p>Note: we recommend not to configure mount-links in <tt>core-site.xml</tt> if you configure above valid path. Otherwise both mount links will be mixed and can lead to a confused behavior.</p> |
| <p>If you copy the <tt>mount-table.<versionNumber>.xml</tt>, you may consider having big replication factor depending on your cluster size. So, that file will be available locally to majority of clients as applications(MR/YARN/HBASE..etc) use locality on HDFS when reading <tt>mount-table.<versionNumber>.xml</tt>.</p></div></div> |
| <div class="section"> |
| <h2><a name="DFSAdmin_commands_with_View_File_System_Overload_Scheme"></a>DFSAdmin commands with View File System Overload Scheme</h2> |
| <p>Please refer to the <a href="./HDFSCommands.html#dfsadmin_with_ViewFsOverloadScheme">HDFSCommands Guide</a></p></div> |
| <div class="section"> |
| <h2><a name="Accessing_paths_without_authority"></a>Accessing paths without authority</h2> |
| <p>Accessing paths like <tt>hdfs:///foo/bar</tt>, <tt>hdfs:/foo/bar</tt> or <tt>viewfs:/foo/bar</tt>, where the authority (cluster name or hostname) of the path is not specified, is very common. This is especially true when the same code is expected to run on multiple clusters with different names or HDFS Namenodes.</p> |
| <p>When <tt>ViewFileSystemOverloadScheme</tt> is used (as described above), and if (a) the scheme of the path being accessed is different from the scheme of the path specified as <tt>fs.defaultFS</tt> and (b) if the path doesn’t have an authority specified, accessing the path can result in an error like <tt>Empty Mount table in config for viewfs://default/</tt>. For example, when the following configuration is used but a path like <tt>viewfs:/foo/bar</tt> or <tt>viewfs:///foo/bar</tt> is accessed, such an error arises.</p> |
| |
| <div> |
| <div> |
| <pre class="source"><property> |
| <name>fs.hdfs.impl</name> |
| <value>org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme</value> |
| </property> |
| |
| <property> |
| <name>fs.defaultFS</name> |
| <value>hdfs://cluster/</value> |
| </property> |
| </pre></div></div> |
| |
| <div class="section"> |
| <h3><a name="Solution"></a>Solution</h3> |
| <p>To avoid the above problem, the configuration <tt>fs.viewfs.mounttable.default.name.key</tt> has to be set to the name of the cluster, i.e, the following should be added to <tt>core-site.xml</tt></p> |
| |
| <div> |
| <div> |
| <pre class="source"><property> |
| <name>fs.viewfs.mounttable.default.name.key</name> |
| <value>cluster</value> |
| </property> |
| </pre></div></div> |
| |
| <p>The string in this configuration <tt>cluster</tt> should match the name of the authority in the value of <tt>fs.defaultFS</tt>. Further, the configuration should have a mount table configured correctly as in the above examples, i.e., the configurations <tt>fs.viewfs.mounttable.*cluster*.link.<mountLinkPath></tt> should be set (note the same string <tt>cluster</tt> is used in these configurations).</p></div></div> |
| <div class="section"> |
| <h2><a name="Appendix:_A_Mount_Table_Configuration_with_XInclude"></a>Appendix: A Mount Table Configuration with XInclude</h2> |
| <p>If users have a HTTP server in trusted network and don’t need authentication mechanism to it, you can also place your mount-table.xml file in that server and configure XInclude xml tag with <tt>mount-table.xml</tt> file.</p> |
| |
| <div> |
| <div> |
| <pre class="source"><configuration xmlns:xi="http://www.w3.org/2001/XInclude"> |
| <xi:include href="http://myserver/mountTable/mountTable.xml" /> |
| </configuration> |
| </pre></div></div> |
| |
| <p>The Apache Hadoop configuration has the capability to read the http urls from XInclude and load into configurations. If you choose this option, please don’t configure mount-table configuration items in <tt>core-site.xml</tt> or at <tt>fs.viewfs.mounttable.path</tt>. Please note, Hadoop configuration XInclude does not use SPNego authentication when opening url. So, this will not work if http server where you placed <tt>mount-table.xml</tt> needs authentication.</p></div> |
| </div> |
| </div> |
| <div class="clear"> |
| <hr/> |
| </div> |
| <div id="footer"> |
| <div class="xright"> |
| © 2008-2021 |
| Apache Software Foundation |
| |
| - <a href="http://maven.apache.org/privacy-policy.html">Privacy Policy</a>. |
| Apache Maven, Maven, Apache, the Apache feather logo, and the Apache Maven project logos are trademarks of The Apache Software Foundation. |
| </div> |
| <div class="clear"> |
| <hr/> |
| </div> |
| </div> |
| </body> |
| </html> |
