blob: 365b60b4fa56e66d709a51ce4d90bca141ab0470 [file] [log] [blame]
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
See the License for the specific language governing permissions and
limitations under the License.
<h1>A client for the Kosmos filesystem (KFS)</h1>
This pages describes how to use Kosmos Filesystem
(<a href=""> KFS </a>) as a backing
store with Hadoop. This page assumes that you have downloaded the
KFS software and installed necessary binaries as outlined in the KFS
<li>In the Hadoop conf directory edit core-site.xml,
add the following:
&lt;description&gt;The FileSystem for kfs: uris.&lt;/description&gt;
<li>In the Hadoop conf directory edit core-site.xml,
adding the following (with appropriate values for
&lt;server&gt; and &lt;port&gt;):
&lt;description&gt;The location of the KFS meta server.&lt;/description&gt;
&lt;description&gt;The location of the meta server's port.&lt;/description&gt;
<li>Copy KFS's <i> kfs-0.1.jar </i> to Hadoop's lib directory. This step
enables Hadoop's to load the KFS specific modules. Note
that, kfs-0.1.jar was built when you compiled KFS source
code. This jar file contains code that calls KFS's client
library code via JNI; the native code is in KFS's <i> </i> library.
<li> When the Hadoop map/reduce trackers start up, those
processes (on local as well as remote nodes) will now need to load
KFS's <i> </i> library. To simplify this process, it is advisable to
store in an NFS accessible directory (similar to where
Hadoop binaries/scripts are stored); then, modify Hadoop's
conf/ adding the following line and providing suitable
value for &lt;path&gt;:
export LD_LIBRARY_PATH=&lt;path&gt;
<li>Start only the map/reduce trackers
<br />
example: execute Hadoop's bin/</li>
If the map/reduce job trackers start up, all file-I/O is done to KFS.