Apache Airavata MFT is a high-performance, multi-protocol data transfer engine to orchestrate data movement and operations across most cloud and On-premises storages. MFT aims to abstract the complexity of heterogenous storages by providing a unified and simple interface for users to seamlessly access and move data across any storage endpoint. To accomplish this goal, MFT provides simple but highly-performing tools to access most cloud and on-premise storages as seamlessly as they access local files in their workstations.
Apache Airavata MFT bundles easily deployable agents that auto determine optimum network path with additional multi-channel, parallel data paths to optimize the transfer performance to gain the maximum throughput between storage endpoints. MFT utilizes parallel Agents to transfer data between endpoints to gain the advantage of multiple network links.
MFT requires Java 11+ and python3.10+ to install Airavata MFT in your environment. MFT currently supports Linux and MacOS operating systems. Contributions to support Windows are welcome!!.
Following commands will download Airavata MFT into your machine and start the MFT service.
pip3 install airavata-mft-cli mft init
If the installer failed for M1 and M2 Macs complaining about grpcio installation. Follow the solution mentioned in here. You might have to uninstall already installed grpcio and grpcio-tools distributions first. For other common installation issues, please refer to the troubleshooting section.
To stop MFT after using
mft stop
First you need to register your storage endpoints into MFT in order to access them. Registering storage is an interactive process and you can easily register those without prior knowledge
mft storage add
This will ask the type of storage you need and credentials to access those storages. To list already added storages, you can run
mft storage list
In Airavata MFT, we provide a unified interface to access the data in any storage. Users can access data in storages just as they access data in their computers. MFT converts user queries into storage specific data representations (POSIX, Block, Objects, ..) internally
mft ls <storage name> mft ls <storage name>/<resource path>
Copying data between storages are simple as copying data between directories of local machine for users. MFT takes care of network path optimizations, parallel data path selections and selections or creations of suitable transfer agents.
mft cp <source storage name>/<path> <destination storage name>/<path>
MFT is capable of auto detecting directory copying and file copying based on the path given.
This is our very first attempt release Airavata MFT for community usage and there might be lots of corner cases that we have not noticed. All the logs of MFT service are available in ~/.mft/Standalone-Service-0.01/logs/airavata.log. If you see any error while using MFT, please report that in our Github issue page and we will respond as soon as possible. We really appreciate your contribution as it will greatly help to improve the stability of the product.
ERROR: Could not find a version that satisfies the requirement airavata-mft-cli (from versions: none) ERROR: No matching distribution found for airavata-mft-cli
If the Error still occurs after installing the right python version, try creating a virtual environemnt
python3.10 -m venv venv source venv/bin/activate pip install airavata-mft-cli