| <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
| <!-- |
| | Generated by Apache Maven Doxia at 2021-06-15 |
| | Rendered using Apache Maven Stylus Skin 1.5 |
| --> |
| <html xmlns="http://www.w3.org/1999/xhtml"> |
| <head> |
| <title>Apache Hadoop 3.3.1 – Testing the Filesystem Contract</title> |
| <style type="text/css" media="all"> |
| @import url("../css/maven-base.css"); |
| @import url("../css/maven-theme.css"); |
| @import url("../css/site.css"); |
| </style> |
| <link rel="stylesheet" href="../css/print.css" type="text/css" media="print" /> |
| <meta name="Date-Revision-yyyymmdd" content="20210615" /> |
| <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> |
| </head> |
| <body class="composite"> |
| <div id="banner"> |
| <a href="http://hadoop.apache.org/" id="bannerLeft"> |
| <img src="http://hadoop.apache.org/images/hadoop-logo.jpg" alt="" /> |
| </a> |
| <a href="http://www.apache.org/" id="bannerRight"> |
| <img src="http://www.apache.org/images/asf_logo_wide.png" alt="" /> |
| </a> |
| <div class="clear"> |
| <hr/> |
| </div> |
| </div> |
| <div id="breadcrumbs"> |
| |
| <div class="xleft"> |
| <a href="http://www.apache.org/" class="externalLink">Apache</a> |
| > |
| <a href="http://hadoop.apache.org/" class="externalLink">Hadoop</a> |
| > |
| <a href="../../index.html">Apache Hadoop Project Dist POM</a> |
| > |
| <a href="../index.html">Apache Hadoop 3.3.1</a> |
| > |
| Testing the Filesystem Contract |
| </div> |
| <div class="xright"> <a href="http://wiki.apache.org/hadoop" class="externalLink">Wiki</a> |
| | |
| <a href="https://gitbox.apache.org/repos/asf/hadoop.git" class="externalLink">git</a> |
| | |
| <a href="http://hadoop.apache.org/" class="externalLink">Apache Hadoop</a> |
| |
| | Last Published: 2021-06-15 |
| | Version: 3.3.1 |
| </div> |
| <div class="clear"> |
| <hr/> |
| </div> |
| </div> |
| <div id="leftColumn"> |
| <div id="navcolumn"> |
| |
| <h5>General</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../index.html">Overview</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/SingleCluster.html">Single Node Setup</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/ClusterSetup.html">Cluster Setup</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/CommandsManual.html">Commands Reference</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/FileSystemShell.html">FileSystem Shell</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/Compatibility.html">Compatibility Specification</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/DownstreamDev.html">Downstream Developer's Guide</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/AdminCompatibilityGuide.html">Admin Compatibility Guide</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/InterfaceClassification.html">Interface Classification</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/filesystem/index.html">FileSystem Specification</a> |
| </li> |
| </ul> |
| <h5>Common</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/CLIMiniCluster.html">CLI Mini Cluster</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/FairCallQueue.html">Fair Call Queue</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/NativeLibraries.html">Native Libraries</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/Superusers.html">Proxy User</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/RackAwareness.html">Rack Awareness</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/SecureMode.html">Secure Mode</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/ServiceLevelAuth.html">Service Level Authorization</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/HttpAuthentication.html">HTTP Authentication</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/CredentialProviderAPI.html">Credential Provider API</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-kms/index.html">Hadoop KMS</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/Tracing.html">Tracing</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/UnixShellGuide.html">Unix Shell Guide</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/registry/index.html">Registry</a> |
| </li> |
| </ul> |
| <h5>HDFS</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsDesign.html">Architecture</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html">User Guide</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HDFSCommands.html">Commands Reference</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html">NameNode HA With QJM</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithNFS.html">NameNode HA With NFS</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/ObserverNameNode.html">Observer NameNode</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/Federation.html">Federation</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/ViewFs.html">ViewFs</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/ViewFsOverloadScheme.html">ViewFsOverloadScheme</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsSnapshots.html">Snapshots</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsEditsViewer.html">Edits Viewer</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsImageViewer.html">Image Viewer</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html">Permissions and HDFS</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html">Quotas and HDFS</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/LibHdfs.html">libhdfs (C API)</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/WebHDFS.html">WebHDFS (REST API)</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-hdfs-httpfs/index.html">HttpFS</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html">Short Circuit Local Reads</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/CentralizedCacheManagement.html">Centralized Cache Management</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html">NFS Gateway</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html">Rolling Upgrade</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/ExtendedAttributes.html">Extended Attributes</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html">Transparent Encryption</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsMultihoming.html">Multihoming</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html">Storage Policies</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/MemoryStorage.html">Memory Storage Support</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/SLGUserGuide.html">Synthetic Load Generator</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html">Erasure Coding</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HDFSDiskbalancer.html">Disk Balancer</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsUpgradeDomain.html">Upgrade Domain</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsDataNodeAdminGuide.html">DataNode Admin</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs-rbf/HDFSRouterFederation.html">Router Federation</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsProvidedStorage.html">Provided Storage</a> |
| </li> |
| </ul> |
| <h5>MapReduce</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html">Tutorial</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredCommands.html">Commands Reference</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html">Compatibility with 1.x</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html">Encrypted Shuffle</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html">Pluggable Shuffle/Sort</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/DistributedCacheDeploy.html">Distributed Cache Deploy</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/SharedCacheSupport.html">Support for YARN Shared Cache</a> |
| </li> |
| </ul> |
| <h5>MapReduce REST APIs</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredAppMasterRest.html">MR Application Master</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-hs/HistoryServerRest.html">MR History Server</a> |
| </li> |
| </ul> |
| <h5>YARN</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/YARN.html">Architecture</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/YarnCommands.html">Commands Reference</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html">Capacity Scheduler</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/FairScheduler.html">Fair Scheduler</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html">ResourceManager Restart</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html">ResourceManager HA</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/ResourceModel.html">Resource Model</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/NodeLabel.html">Node Labels</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/NodeAttributes.html">Node Attributes</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/WebApplicationProxy.html">Web Application Proxy</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/TimelineServer.html">Timeline Server</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html">Timeline Service V.2</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html">Writing YARN Applications</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html">YARN Application Security</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/NodeManager.html">NodeManager</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/DockerContainers.html">Running Applications in Docker Containers</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/RuncContainers.html">Running Applications in runC Containers</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/NodeManagerCgroups.html">Using CGroups</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/SecureContainer.html">Secure Containers</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/ReservationSystem.html">Reservation System</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/GracefulDecommission.html">Graceful Decommission</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html">Opportunistic Containers</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/Federation.html">YARN Federation</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/SharedCache.html">Shared Cache</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/UsingGpus.html">Using GPU</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/UsingFPGA.html">Using FPGA</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html">Placement Constraints</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/YarnUI2.html">YARN UI2</a> |
| </li> |
| </ul> |
| <h5>YARN REST APIs</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html">Introduction</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html">Resource Manager</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/NodeManagerRest.html">Node Manager</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/TimelineServer.html#Timeline_Server_REST_API_v1">Timeline Server</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html#Timeline_Service_v.2_REST_API">Timeline Service V.2</a> |
| </li> |
| </ul> |
| <h5>YARN Service</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/yarn-service/Overview.html">Overview</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/yarn-service/QuickStart.html">QuickStart</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/yarn-service/Concepts.html">Concepts</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/yarn-service/YarnServiceAPI.html">Yarn Service API</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/yarn-service/ServiceDiscovery.html">Service Discovery</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/yarn-service/SystemServices.html">System Services</a> |
| </li> |
| </ul> |
| <h5>Hadoop Compatible File Systems</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-aliyun/tools/hadoop-aliyun/index.html">Aliyun OSS</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-aws/tools/hadoop-aws/index.html">Amazon S3</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-azure/index.html">Azure Blob Storage</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-azure-datalake/index.html">Azure Data Lake Storage</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-openstack/index.html">OpenStack Swift</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-cos/cloud-storage/index.html">Tencent COS</a> |
| </li> |
| </ul> |
| <h5>Auth</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-auth/index.html">Overview</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-auth/Examples.html">Examples</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-auth/Configuration.html">Configuration</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-auth/BuildingIt.html">Building</a> |
| </li> |
| </ul> |
| <h5>Tools</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-streaming/HadoopStreaming.html">Hadoop Streaming</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-archives/HadoopArchives.html">Hadoop Archives</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-archive-logs/HadoopArchiveLogs.html">Hadoop Archive Logs</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-distcp/DistCp.html">DistCp</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-gridmix/GridMix.html">GridMix</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-rumen/Rumen.html">Rumen</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-resourceestimator/ResourceEstimator.html">Resource Estimator Service</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-sls/SchedulerLoadSimulator.html">Scheduler Load Simulator</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/Benchmarking.html">Hadoop Benchmarking</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-dynamometer/Dynamometer.html">Dynamometer</a> |
| </li> |
| </ul> |
| <h5>Reference</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/release/">Changelog and Release Notes</a> |
| </li> |
| <li class="none"> |
| <a href="../../../api/index.html">Java API docs</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/UnixShellAPI.html">Unix Shell API</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/Metrics.html">Metrics</a> |
| </li> |
| </ul> |
| <h5>Configuration</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/core-default.xml">core-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/hdfs-default.xml">hdfs-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs-rbf/hdfs-rbf-default.xml">hdfs-rbf-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml">mapred-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-common/yarn-default.xml">yarn-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-kms/kms-default.html">kms-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-hdfs-httpfs/httpfs-default.html">httpfs-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/DeprecatedProperties.html">Deprecated Properties</a> |
| </li> |
| </ul> |
| <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"> |
| <img alt="Built by Maven" src="../images/logos/maven-feather.png"/> |
| </a> |
| |
| </div> |
| </div> |
| <div id="bodyColumn"> |
| <div id="contentBox"> |
| <!--- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. See accompanying LICENSE file. |
| --> |
| <h1>Testing the Filesystem Contract</h1> |
| <ul> |
| <li><a href="#Running_the_tests">Running the tests</a> |
| <ul> |
| <li><a href="#ftp:.2F.2F">ftp://</a></li> |
| <li><a href="#swift:.2F.2F">swift://</a></li></ul></li> |
| <li><a href="#Testing_a_new_filesystem">Testing a new filesystem</a> |
| <ul> |
| <li><a href="#Handling_test_failures">Handling test failures</a></li> |
| <li><a href="#a.E2.80.98Lax_vs_Strict.E2.80.99_exceptions">‘Lax vs Strict’ exceptions</a></li> |
| <li><a href="#Supporting_FileSystems_with_login_and_authentication_parameters">Supporting FileSystems with login and authentication parameters</a></li> |
| <li><a href="#Important:_passing_the_tests_does_not_guarantee_compatibility">Important: passing the tests does not guarantee compatibility</a></li></ul></li> |
| <li><a href="#Adding_a_new_test_suite">Adding a new test suite</a> |
| <ul> |
| <li><a href="#Root_manipulation_tests">Root manipulation tests</a></li> |
| <li><a href="#Scalability_tests">Scalability tests</a></li></ul></li> |
| <li><a href="#Extending_the_specification">Extending the specification</a></li></ul> |
| |
| <div class="section"> |
| <h2><a name="Running_the_tests"></a>Running the tests</h2> |
| <p>A normal Hadoop test run will test those FileSystems that can be tested locally via the local filesystem. This typically means <tt>file://</tt> and its underlying <tt>LocalFileSystem</tt>, and <tt>hdfs://</tt> via the HDFS MiniCluster.</p> |
| <p>Other filesystems are skipped unless there is a specific configuration to the remote server providing the filesystem.</p> |
| <p>These filesystem bindings must be defined in an XML configuration file, usually <tt>hadoop-common-project/hadoop-common/src/test/resources/contract-test-options.xml</tt>. This file is excluded and should not be checked in.</p> |
| <div class="section"> |
| <h3><a name="ftp:.2F.2F"></a>ftp://</h3> |
| <p>In <tt>contract-test-options.xml</tt>, the filesystem name must be defined in the property <tt>fs.contract.test.fs.ftp</tt>. The specific login options to connect to the FTP Server must then be provided.</p> |
| <p>A path to a test directory must also be provided in the option <tt>fs.contract.test.ftp.testdir</tt>. This is the directory under which operations take place.</p> |
| <p>Example:</p> |
| |
| <div> |
| <div> |
| <pre class="source"><configuration> |
| <property> |
| <name>fs.contract.test.fs.ftp</name> |
| <value>ftp://server1/</value> |
| </property> |
| |
| <property> |
| <name>fs.ftp.user.server1</name> |
| <value>testuser</value> |
| </property> |
| |
| <property> |
| <name>fs.contract.test.ftp.testdir</name> |
| <value>/home/testuser/test</value> |
| </property> |
| |
| <property> |
| <name>fs.ftp.password.server1</name> |
| <value>secret-login</value> |
| </property> |
| </configuration> |
| </pre></div></div> |
| </div> |
| <div class="section"> |
| <h3><a name="swift:.2F.2F"></a>swift://</h3> |
| <p>The OpenStack Swift login details must be defined in the file <tt>/hadoop-tools/hadoop-openstack/src/test/resources/contract-test-options.xml</tt>. The standard hadoop-common <tt>contract-test-options.xml</tt> resource file cannot be used, as that file does not get included in <tt>hadoop-common-test.jar</tt>.</p> |
| <p>In <tt>/hadoop-tools/hadoop-openstack/src/test/resources/contract-test-options.xml</tt> the Swift bucket name must be defined in the property <tt>fs.contract.test.fs.swift</tt>, along with the login details for the specific Swift service provider in which the bucket is posted.</p> |
| |
| <div> |
| <div> |
| <pre class="source"><configuration> |
| <property> |
| <name>fs.contract.test.fs.swift</name> |
| <value>swift://swiftbucket.rackspace/</value> |
| </property> |
| |
| <property> |
| <name>fs.swift.service.rackspace.auth.url</name> |
| <value>https://auth.api.rackspacecloud.com/v2.0/tokens</value> |
| <description>Rackspace US (multiregion)</description> |
| </property> |
| |
| <property> |
| <name>fs.swift.service.rackspace.username</name> |
| <value>this-is-your-username</value> |
| </property> |
| |
| <property> |
| <name>fs.swift.service.rackspace.region</name> |
| <value>DFW</value> |
| </property> |
| |
| <property> |
| <name>fs.swift.service.rackspace.apikey</name> |
| <value>ab0bceyoursecretapikeyffef</value> |
| </property> |
| |
| </configuration> |
| </pre></div></div> |
| |
| <ol style="list-style-type: decimal"> |
| |
| <li>Often the different public cloud Swift infrastructures exhibit different behaviors (authentication and throttling in particular). We recommand that testers create accounts on as many of these providers as possible and test against each of them.</li> |
| <li>They can be slow, especially remotely. Remote links are also the most likely to make eventual-consistency behaviors visible, which is a mixed benefit.</li> |
| </ol></div></div> |
| <div class="section"> |
| <h2><a name="Testing_a_new_filesystem"></a>Testing a new filesystem</h2> |
| <p>The core of adding a new FileSystem to the contract tests is adding a new contract class, then creating a new non-abstract test class for every test suite that you wish to test.</p> |
| <ol style="list-style-type: decimal"> |
| |
| <li>Do not try and add these tests into Hadoop itself. They won’t be added to the source tree. The tests must live with your own filesystem source.</li> |
| <li>Create a package in your own test source tree (usually) under <tt>contract</tt>, for the files and tests.</li> |
| <li>Subclass <tt>AbstractFSContract</tt> for your own contract implementation.</li> |
| <li>For every test suite you plan to support create a non-abstract subclass, with the name starting with <tt>Test</tt> and the name of the filesystem. Example: <tt>TestHDFSRenameContract</tt>.</li> |
| <li>These non-abstract classes must implement the abstract method <tt>createContract()</tt>.</li> |
| <li>Identify and document any filesystem bindings that must be defined in a <tt>src/test/resources/contract-test-options.xml</tt> file of the specific project.</li> |
| <li>Run the tests until they work.</li> |
| </ol> |
| <p>As an example, here is the implementation of the test of the <tt>create()</tt> tests for the local filesystem.</p> |
| |
| <div> |
| <div> |
| <pre class="source">package org.apache.hadoop.fs.contract.localfs; |
| |
| import org.apache.hadoop.conf.Configuration; |
| import org.apache.hadoop.fs.contract.AbstractCreateContractTest; |
| import org.apache.hadoop.fs.contract.AbstractFSContract; |
| |
| public class TestLocalCreateContract extends AbstractCreateContractTest { |
| @Override |
| protected AbstractFSContract createContract(Configuration conf) { |
| return new LocalFSContract(conf); |
| } |
| } |
| </pre></div></div> |
| |
| <p>The standard implementation technique for subclasses of <tt>AbstractFSContract</tt> is to be driven entirely by a Hadoop XML configuration file stored in the test resource tree. The best practise is to store it under <tt>/contract</tt> with the name of the FileSystem, such as <tt>contract/localfs.xml</tt>. Having the XML file define all FileSystem options makes the listing of FileSystem behaviors immediately visible.</p> |
| <p>The <tt>LocalFSContract</tt> is a special case of this, as it must adjust its case sensitivity policy based on the OS on which it is running: for both Windows and OS/X, the filesystem is case insensitive, so the <tt>ContractOptions.IS_CASE_SENSITIVE</tt> option must be set to false. Furthermore, the Windows filesystem does not support Unix file and directory permissions, so the relevant flag must also be set. This is done <i>after</i> loading the XML contract file from the resource tree, simply by updating the now-loaded configuration options:</p> |
| |
| <div> |
| <div> |
| <pre class="source"> getConf().setBoolean(getConfKey(ContractOptions.SUPPORTS_UNIX_PERMISSIONS), false); |
| </pre></div></div> |
| |
| <div class="section"> |
| <h3><a name="Handling_test_failures"></a>Handling test failures</h3> |
| <p>If your new <tt>FileSystem</tt> test cases fails one of the contract tests, what you can you do?</p> |
| <p>It depends on the cause of the problem</p> |
| <ol style="list-style-type: decimal"> |
| |
| <li>Case: custom <tt>FileSystem</tt> subclass class doesn’t correctly implement specification. Fix.</li> |
| <li>Case: Underlying filesystem doesn’t behave in a way that matches Hadoop’s expectations. Ideally, fix. Or try to make your <tt>FileSystem</tt> subclass hide the differences, e.g. by translating exceptions.</li> |
| <li>Case: fundamental architectural differences between your filesystem and Hadoop. Example: different concurrency and consistency model. Recommendation: document and make clear that the filesystem is not compatible with HDFS.</li> |
| <li>Case: test does not match the specification. Fix: patch test, submit the patch to Hadoop.</li> |
| <li>Case: specification incorrect. The underlying specification is (with a few exceptions) HDFS. If the specification does not match HDFS, HDFS should normally be assumed to be the real definition of what a FileSystem should do. If there’s a mismatch, please raise it on the <tt>hdfs-dev</tt> mailing list. Note that while FileSystem tests live in the core Hadoop codebase, it is the HDFS team who owns the FileSystem specification and the tests that accompany it.</li> |
| </ol> |
| <p>If a test needs to be skipped because a feature is not supported, look for a existing configuration option in the <tt>ContractOptions</tt> class. If there is no method, the short term fix is to override the method and use the <tt>ContractTestUtils.skip()</tt> message to log the fact that a test is skipped. Using this method prints the message to the logs, then tells the test runner that the test was skipped. This highlights the problem.</p> |
| <p>A recommended strategy is to call the superclass, catch the exception, and verify that the exception class and part of the error string matches that raised by the current implementation. It should also <tt>fail()</tt> if superclass actually succeeded -that is it failed the way that the implemention does not currently do. This will ensure that the test path is still executed, any other failure of the test -possibly a regression- is picked up. And, if the feature does become implemented, that the change is picked up.</p> |
| <p>A long-term solution is to enhance the base test to add a new optional feature key. This will require collaboration with the developers on the <tt>hdfs-dev</tt> mailing list.</p></div> |
| <div class="section"> |
| <h3><a name="a.E2.80.98Lax_vs_Strict.E2.80.99_exceptions"></a>‘Lax vs Strict’ exceptions</h3> |
| <p>The contract tests include the notion of strict vs lax exceptions. <i>Strict</i> exception reporting means: reports failures using specific subclasses of <tt>IOException</tt>, such as <tt>FileNotFoundException</tt>, <tt>EOFException</tt> and so on. <i>Lax</i> reporting means throws <tt>IOException</tt>.</p> |
| <p>While FileSystems SHOULD raise stricter exceptions, there may be reasons why they cannot. Raising lax exceptions is still allowed, it merely hampers diagnostics of failures in user applications. To declare that a FileSystem does not support the stricter exceptions, set the option <tt>fs.contract.supports-strict-exceptions</tt> to false.</p></div> |
| <div class="section"> |
| <h3><a name="Supporting_FileSystems_with_login_and_authentication_parameters"></a>Supporting FileSystems with login and authentication parameters</h3> |
| <p>Tests against remote FileSystems will require the URL to the FileSystem to be specified; tests against remote FileSystems that require login details require usernames/IDs and passwords.</p> |
| <p>All these details MUST be required to be placed in the file <tt>src/test/resources/contract-test-options.xml</tt>, and your SCM tools configured to never commit this file to subversion, git or equivalent. Furthermore, the build MUST be configured to never bundle this file in any <tt>-test</tt> artifacts generated. The Hadoop build does this, excluding <tt>src/test/**/*.xml</tt> from the JAR files. In addition, <tt>src/test/resources/auth-keys.xml</tt> will need to be created. It can be a copy of <tt>contract-test-options.xml</tt>. The <tt>AbstractFSContract</tt> class automatically loads this resource file if present; specific keys for specific test cases can be added.</p> |
| <p>As an example, here are what S3A test keys look like:</p> |
| |
| <div> |
| <div> |
| <pre class="source"><configuration> |
| <property> |
| <name>fs.contract.test.fs.s3a</name> |
| <value>s3a://tests3contract</value> |
| </property> |
| |
| <property> |
| <name>fs.s3a.access.key</name> |
| <value>DONOTPCOMMITTHISKEYTOSCM</value> |
| </property> |
| |
| <property> |
| <name>fs.s3a.secret.key</name> |
| <value>DONOTEVERSHARETHISSECRETKEY!</value> |
| </property> |
| </configuration> |
| </pre></div></div> |
| |
| <p>The <tt>AbstractBondedFSContract</tt> automatically skips a test suite if the FileSystem URL is not defined in the property <tt>fs.contract.test.fs.%s</tt>, where <tt>%s</tt> matches the schema name of the FileSystem.</p> |
| <p>When running the tests <tt>maven.test.skip</tt> will need to be turned off since it is true by default on these tests. This can be done with a command like <tt>mvn test -Ptests-on</tt>.</p></div> |
| <div class="section"> |
| <h3><a name="Important:_passing_the_tests_does_not_guarantee_compatibility"></a>Important: passing the tests does not guarantee compatibility</h3> |
| <p>Passing all the FileSystem contract tests does not mean that a filesystem can be described as “compatible with HDFS”. The tests try to look at the isolated functionality of each operation, and focus on the preconditions and postconditions of each action. Core areas not covered are concurrency and aspects of failure across a distributed system.</p> |
| <ul> |
| |
| <li>Consistency: are all changes immediately visible?</li> |
| <li>Atomicity: are operations which HDFS guarantees to be atomic equally so on the new filesystem.</li> |
| <li>Idempotency: if the filesystem implements any retry policy, is idempotent even while other clients manipulate the filesystem?</li> |
| <li>Scalability: does it support files as large as HDFS, or as many in a single directory?</li> |
| <li>Durability: do files actually last -and how long for?</li> |
| </ul> |
| <p>Proof that this is is true is the fact that the Amazon S3 and OpenStack Swift object stores are eventually consistent object stores with non-atomic rename and delete operations. Single threaded test cases are unlikely to see some of the concurrency issues, while consistency is very often only visible in tests that span a datacenter.</p> |
| <p>There are also some specific aspects of the use of the FileSystem API:</p> |
| <ul> |
| |
| <li>Compatibility with the <tt>hadoop -fs</tt> CLI.</li> |
| <li>Whether the blocksize policy produces file splits that are suitable for analytics workss. (as an example, a blocksize of 1 matches the specification, but as it tells MapReduce jobs to work a byte at a time, unusable).</li> |
| </ul> |
| <p>Tests that verify these behaviors are of course welcome.</p></div></div> |
| <div class="section"> |
| <h2><a name="Adding_a_new_test_suite"></a>Adding a new test suite</h2> |
| <ol style="list-style-type: decimal"> |
| |
| <li>New tests should be split up with a test class per operation, as is done for <tt>seek()</tt>, <tt>rename()</tt>, <tt>create()</tt>, and so on. This is to match up the way that the FileSystem contract specification is split up by operation. It also makes it easier for FileSystem implementors to work on one test suite at a time.</li> |
| <li>Subclass <tt>AbstractFSContractTestBase</tt> with a new abstract test suite class. Again, use <tt>Abstract</tt> in the title.</li> |
| <li>Look at <tt>org.apache.hadoop.fs.contract.ContractTestUtils</tt> for utility classes to aid testing, with lots of filesystem-centric assertions. Use these to make assertions about the filesystem state, and to incude diagnostics information such as directory listings and dumps of mismatched files when an assertion actually fails.</li> |
| <li>Write tests for the local, raw local and HDFS filesystems -if one of these fails the tests then there is a sign of a problem -though be aware that they do have differnces</li> |
| <li>Test on the object stores once the core filesystems are passing the tests.</li> |
| <li>Try and log failures with as much detail as you can -the people debugging the failures will appreciate it.</li> |
| </ol> |
| <div class="section"> |
| <h3><a name="Root_manipulation_tests"></a>Root manipulation tests</h3> |
| <p>Some tests work directly against the root filesystem, attempting to do things like rename “/” and similar actions. The root directory is “special”, and it’s important to test this, especially on non-POSIX filesystems such as object stores. These tests are potentially very destructive to native filesystems, so use care.</p> |
| <ol style="list-style-type: decimal"> |
| |
| <li> |
| |
| <p>Add the tests under <tt>AbstractRootDirectoryContractTest</tt> or create a new test with (a) <tt>Root</tt> in the title and (b) a check in the setup method to skip the test if root tests are disabled:</p> |
| |
| <div> |
| <div> |
| <pre class="source"> skipIfUnsupported(TEST_ROOT_TESTS_ENABLED); |
| </pre></div></div> |
| </li> |
| <li> |
| |
| <p>Don’t provide an implementation of this test suite to run against the local FS.</p> |
| </li> |
| </ol></div> |
| <div class="section"> |
| <h3><a name="Scalability_tests"></a>Scalability tests</h3> |
| <p>Tests designed to generate scalable load -and that includes a large number of small files, as well as fewer larger files, should be designed to be configurable, so that users of the test suite can configure the number and size of files.</p> |
| <p>Be aware that on object stores, the directory rename operation is usually <tt>O(files)*O(data)</tt> while the delete operation is <tt>O(files)</tt>. The latter means even any directory cleanup operations may take time and can potentially timeout. It is important to design tests that work against remote filesystems with possible delays in all operations.</p></div></div> |
| <div class="section"> |
| <h2><a name="Extending_the_specification"></a>Extending the specification</h2> |
| <p>The specification is incomplete. It doesn’t have complete coverage of the FileSystem classes, and there may be bits of the existing specified classes that are not covered.</p> |
| <ol style="list-style-type: decimal"> |
| |
| <li>Look at the implementations of a class/interface/method to see what they do, especially HDFS and local. These are the documentation of what is done today.</li> |
| <li>Look at the POSIX API specification.</li> |
| <li>Search through the HDFS JIRAs for discussions on FileSystem topics, and try to understand what was meant to happen, as well as what does happen.</li> |
| <li>Use an IDE to find out how methods are used in Hadoop, HBase and other parts of the stack. Although this assumes that these are representative Hadoop applications, it will at least show how applications <i>expect</i> a FileSystem to behave.</li> |
| <li>Look in the java.io source to see how the bunded FileSystem classes are expected to behave -and read their javadocs carefully.</li> |
| <li>If something is unclear -as on the hdfs-dev list.</li> |
| <li>Don’t be afraid to write tests to act as experiments and clarify what actually happens. Use the HDFS behaviours as the normative guide.</li> |
| </ol></div> |
| </div> |
| </div> |
| <div class="clear"> |
| <hr/> |
| </div> |
| <div id="footer"> |
| <div class="xright"> |
| © 2008-2021 |
| Apache Software Foundation |
| |
| - <a href="http://maven.apache.org/privacy-policy.html">Privacy Policy</a>. |
| Apache Maven, Maven, Apache, the Apache feather logo, and the Apache Maven project logos are trademarks of The Apache Software Foundation. |
| </div> |
| <div class="clear"> |
| <hr/> |
| </div> |
| </div> |
| </body> |
| </html> |
