blob: a5179440657e27a8be8717ddcc5c6c882a0d35f9 [file] [log] [blame]
<!--
▄▄▄ ██▓███ ▄▄▄ ▄████▄ ██░ ██ ▓█████ ██▓ ▄████ ███▄ █ ██▓▄▄▄█████▓▓█████
▒████▄ ▓██░ ██▒▒████▄ ▒██▀ ▀█ ▓██░ ██▒▓█ ▀ ▓██▒ ██▒ ▀█▒ ██ ▀█ █ ▓██▒▓ ██▒ ▓▒▓█ ▀
▒██ ▀█▄ ▓██░ ██▓▒▒██ ▀█▄ ▒▓█ ▄ ▒██▀▀██░▒███ ▒██▒▒██░▄▄▄░▓██ ▀█ ██▒▒██▒▒ ▓██░ ▒░▒███
░██▄▄▄▄██ ▒██▄█▓▒ ▒░██▄▄▄▄██ ▒▓▓▄ ▄██▒░▓█ ░██ ▒▓█ ▄ ░██░░▓█ ██▓▓██▒ ▐▌██▒░██░░ ▓██▓ ░ ▒▓█ ▄
▓█ ▓██▒▒██▒ ░ ░ ▓█ ▓██▒▒ ▓███▀ ░░▓█▒░██▓░▒████▒ ░██░░▒▓███▀▒▒██░ ▓██░░██░ ▒██▒ ░ ░▒████▒
▒▒ ▓▒█░▒▓▒░ ░ ░ ▒▒ ▓▒█░░ ░▒ ▒ ░ ▒ ░░▒░▒░░ ▒░ ░ ░▓ ░▒ ▒ ░ ▒░ ▒ ▒ ░▓ ▒ ░░ ░░ ▒░ ░
▒ ▒▒ ░░▒ ░ ▒ ▒▒ ░ ░ ▒ ▒ ░▒░ ░ ░ ░ ░ ▒ ░ ░ ░ ░ ░░ ░ ▒░ ▒ ░ ░ ░ ░ ░
░ ▒ ░░ ░ ▒ ░ ░ ░░ ░ ░ ▒ ░░ ░ ░ ░ ░ ░ ▒ ░ ░ ░
░ ░ ░ ░░ ░ ░ ░ ░ ░ ░ ░ ░ ░ ░ ░ ░
-->
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!DOCTYPE html>
<html>
<head>
<link rel="canonical" href="https://ignite.apache.org/features/igfs.html" />
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta http-equiv="Cache-Control" content="no-cache, no-store, must-revalidate" />
<meta http-equiv="Pragma" content="no-cache" />
<meta http-equiv="Expires" content="0" />
<title>In-Memory File System - Apache Ignite</title>
<link media="all" rel="stylesheet" href="/css/all.css?v=1514336028">
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.2.0/css/font-awesome.css" rel="stylesheet">
<link href='https://fonts.googleapis.com/css?family=Open+Sans:400,300,300italic,400italic,600,600italic,700,700italic,800,800italic' rel='stylesheet' type='text/css'>
<!--#include virtual="/includes/sh.html" -->
</head>
<body>
<div id="wrapper">
<!--#include virtual="/includes/header.html" -->
<main id="main" role="main" class="container">
<section id="igfs" class="page-section">
<h1 class="first">In-Memory File System</h1>
<div class="col-sm-12 col-md-12 col-xs-12" style="padding:0 0 10px 0;">
<div class="col-sm-6 col-md-6 col-xs-12" style="padding-left:0; padding-right:0">
<p>
One of the unique capabilities of Apache Ignite is a distributed in-memory file system called
Ignite File System (IGFS). IGFS delivers a similar functionality to Hadoop HDFS, but
only in-memory. In fact, in addition to its own APIs, IGFS implements Hadoop FileSystem API
and can be transparently plugged into Hadoop or Spark deployments.
</p>
<p>
IGFS splits the data from each file into separate data blocks and stores them in
a distributed in-memory cache. However, unlike Hadoop HDFS, IGFS does not need a name node
and automatically determines file data locality using a hashing function.
</p>
</div>
<div class="col-sm-6 col-md-6 col-xs-12" style="padding-right:0">
<img class="img-responsive" src="/images/spark-igfs.png" width="540px" style="float:right;"/>
</div>
</div>
<p>
IGFS can be deployed stand alone, as well as on top of HDFS in which case it becomes
a transparent caching layer for the files stored in HDFS.
</p>
<p>IGFS integrates with native Apache Hadoop distribution, Cloudera CDH, and Hortonworks HDP.</p>
<div class="page-heading">Tachyon Replacement</div>
<p>
IGFS can transparently replace Tachyon file system in Spark deployments. Given that IGFS is based
on battle-tested Ignite data grid technology, it exhibits much better write and read performance
than Tachyon, and is more stable.
</p>
<div class="page-heading">Hadoop File System</div>
<p>
See <a href="http://apacheignite-fs.readme.io/docs/file-system" target="docs">Hadoop integration documentation</a> if
you plan to use IGFS as Hadoop file system. In this case working with IGFS is no different than working
with HDFS.
</p>
<div class="page-heading">GitHub Native API Examples</div>
<p>
Also see <a href="https://github.com/apache/ignite/tree/master/examples/src/main/java/org/apache/ignite/examples/igfs" target="github">IGFS native API examples</a>
available on GitHub.
</p>
</section>
<section id="key-features" class="page-section">
<h2>Ignite File System Features</h2>
<table class="formatted" name="Ignite File System Features">
<thead>
<tr>
<th width="35%" class="left">Feature</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td class="left">On-Heap and Off-Heap</td>
<td>
<p>
IGFS allows to store files either on-heap or off-heap. For larger memory spaces
it is critical to use off-heap to avoid JVM lengthy garbage collection pauses.
</p>
</td>
</tr>
<tr>
<td class="left">IGFS as Hadoop FileSystem</td>
<td>
<p>
IGFS implements Hadoop <code>FileSystem</code> API and can be deployed as a native
Hadoop file system, just like HDFS. This allows to natively deploy IGFS into Hadoop or
Spark installations in a plug-n-play fashion.
</p>
<div class="page-links">
<a href="http://apacheignite-fs.readme.io/docs/file-system" target="docs">Docs for this Feature <i class="fa fa-angle-double-right"></i></a>
</div>
</td>
</tr>
<tr>
<td class="left">Hadoop FileSystem Cache</td>
<td>
<p>
IGFS can also be deployed as a <code>caching layer</code> over another Hadoop File System. In this
case, if a file is updated IGFS, then update will automatically be written through to
HDFS. Also, if a file is read and is not currently in IGFS, it will be automatically
loaded from HDFS.
</p>
<div class="page-links">
<a href="http://apacheignite-fs.readme.io/docs/secondary-file-system" target="docs">Docs for this Feature <i class="fa fa-angle-double-right"></i></a>
</div>
</td>
</tr>
<tr>
<td class="left">Any Hadoop Distribution</td>
<td>
<p>
IGFS integrates with native Apache Hadoop distribution, as well as Cloudera CDH, and
Hortonworks HDP.
</p>
<div class="page-links" style="line-height: 1.3;">
<a href="http://apacheignite-fs.readme.io/docs/installing-on-apache-hadoop" target="docs">IGFS on Apache Hadoop <i class="fa fa-angle-double-right"></i></a><br/>
<a href="http://apacheignite-fs.readme.io/docs/installing-on-cloudera-cdh" target="docs">IGFS on Cloudera CDH <i class="fa fa-angle-double-right"></i></a><br/>
<a href="http://apacheignite-fs.readme.io/docs/installing-on-hortonworks-hdp" target="docs">IGFS on Hortonworks HDP <i class="fa fa-angle-double-right"></i></a>
</div>
</td>
</tr>
</tbody>
</table>
</section>
</main>
<!--#include virtual="/includes/footer.html" -->
</div>
<!--#include virtual="/includes/scripts.html" -->
</body>
</html>