This project is a wrapper around Hadoop's swift native file system. By pulling a smaller dependency tree and shading all dependencies away, this keeps the appearance of Flink being Hadoop-free, from a dependency perspective.
We also relocate the shaded Hadoop version to allow running in a different setup. For this to work, however, we needed to adapt Hadoop's Configuration
class to load a (shaded) core-default-shaded.xml
configuration with the relocated class names of classes loaded via reflection (in the future, we may need to extend this to mapred-default.xml
and hdfs-defaults.xml
and their respective configuration classes).
If you want to change the Hadoop version this project depends on, the following steps are required to keep the shading correct:
openstackhadoop.hadoop.version
property our pom.xml
),org/apache/hadoop/conf/Configuration.java
to src/main/java/org/apache/hadoop/conf/
andcore-default.xml
with core-default-shaded.xml
.org/apache/hadoop/util/NativeCodeLoader.java
to src/main/java/org/apache/hadoop/util/
andstatic { LOG.info("Skipping native-hadoop library for flink-openstack-fs-hadoop's relocated Hadoop... " + "using builtin-java classes where applicable"); }
core-default.xml
to src/main/resources/core-default-shaded.xml
andorg.apache.hadoop
into org.apache.flink.fs.openstackhadoop.shaded.org.apache.hadoop
core-site.xml
to src/test/resources/core-site.xml
(as is)org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem
org.apache.flink.fs.openstackhadoop.shaded
META-INF/services/org.apache.flink.core.fs.FileSystemFactory
file pointing to the org.apache.flink.fs.openstackhadoop.SwiftFileSystemFactory
classMETA-INF/services
should have their names and contents in the relocated org.apache.flink.fs.swifthadoop.shaded
packagecore-default-shaded.xml
filecore-default.xml
or core-site.xml
file