| <html> |
| |
| <!-- |
| Licensed to the Apache Software Foundation (ASF) under one or more |
| contributor license agreements. See the NOTICE file distributed with |
| this work for additional information regarding copyright ownership. |
| The ASF licenses this file to You under the Apache License, Version 2.0 |
| (the "License"); you may not use this file except in compliance with |
| the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| |
| <head></head> |
| <body> |
| <h1>A client for the Kosmos filesystem (KFS)</h1> |
| |
| <h3>Introduction</h3> |
| |
| This pages describes how to use Kosmos Filesystem |
| (<a href="http://kosmosfs.sourceforge.net"> KFS </a>) as a backing |
| store with Hadoop. This page assumes that you have downloaded the |
| KFS software and installed necessary binaries as outlined in the KFS |
| documentation. |
| |
| <h3>Steps</h3> |
| |
| <ul> |
| <li>In the Hadoop conf directory edit core-site.xml, |
| add the following: |
| <pre> |
| <property> |
| <name>fs.kfs.impl</name> |
| <value>org.apache.hadoop.fs.kfs.KosmosFileSystem</value> |
| <description>The FileSystem for kfs: uris.</description> |
| </property> |
| </pre> |
| |
| <li>In the Hadoop conf directory edit core-site.xml, |
| adding the following (with appropriate values for |
| <server> and <port>): |
| <pre> |
| <property> |
| <name>fs.default.name</name> |
| <value>kfs://<server:port></value> |
| </property> |
| |
| <property> |
| <name>fs.kfs.metaServerHost</name> |
| <value><server></value> |
| <description>The location of the KFS meta server.</description> |
| </property> |
| |
| <property> |
| <name>fs.kfs.metaServerPort</name> |
| <value><port></value> |
| <description>The location of the meta server's port.</description> |
| </property> |
| |
| </pre> |
| </li> |
| |
| <li>Copy KFS's <i> kfs-0.1.jar </i> to Hadoop's lib directory. This step |
| enables Hadoop's to load the KFS specific modules. Note |
| that, kfs-0.1.jar was built when you compiled KFS source |
| code. This jar file contains code that calls KFS's client |
| library code via JNI; the native code is in KFS's <i> |
| libkfsClient.so </i> library. |
| </li> |
| |
| <li> When the Hadoop map/reduce trackers start up, those |
| processes (on local as well as remote nodes) will now need to load |
| KFS's <i> libkfsClient.so </i> library. To simplify this process, it is advisable to |
| store libkfsClient.so in an NFS accessible directory (similar to where |
| Hadoop binaries/scripts are stored); then, modify Hadoop's |
| conf/hadoop-env.sh adding the following line and providing suitable |
| value for <path>: |
| <pre> |
| export LD_LIBRARY_PATH=<path> |
| </pre> |
| |
| |
| <li>Start only the map/reduce trackers |
| <br /> |
| example: execute Hadoop's bin/start-mapred.sh</li> |
| </ul> |
| <br/> |
| |
| If the map/reduce job trackers start up, all file-I/O is done to KFS. |
| |
| </body> |
| </html> |
