This project bundles the minimal dependencies from Hadoop's FileSystem abstraction and shades them to avoid dependency conflicts.
This project is the basis for the bundled File System adapters that are based on Hadoop code, but keep the appearance of Flink being Hadoop-free, from a dependency perspective.
For this to work, however, we needed to adapt Hadoop's Configuration
class to load a (shaded) core-default-shaded.xml
configuration with the relocated class names of classes loaded via reflection.
If you want to change the Hadoop version this project depends on, the following steps are required to keep the shading correct:
org/apache/hadoop/conf/Configuration.java
to src/main/java/org/apache/hadoop/conf/
andcore-default.xml
with core-default-shaded.xml
.org/apache/hadoop/util/NativeCodeLoader.java
to src/main/java/org/apache/hadoop/util/
andcore-default.xml
to src/main/resources/core-default-shaded.xml
andorg.apache.hadoop
into org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop
core-site.xml
to src/test/resources/core-site.xml
(as is)org.apache.flink.fs.shaded.hadoop3
META-INF/services/org.apache.flink.core.fs.FileSystemFactory
file pointing to two classes: org.apache.flink.fs.s3hadoop.S3FileSystemFactory
and org.apache.flink.fs.s3hadoop.S3AFileSystemFactory
META-INF/services
should have their names and contents in the relocated org.apache.flink.fs.s3hadoop.shaded
packagecore-default-shaded.xml
filecore-default.xml
or core-site.xml
file