IMPALA-10080: Skip loading HDFS cache pools for non-HDFS file systems
In global invalidate metadata, we always load HDFS cache pools using the
CachePoolReader. Actually, it only works for HDFS file systems, not for
other systems like S3 or local, etc. We already handle this in
CatalogServiceCatalog#CatalogServiceCatalog(). This patch adds a check
in CatalogServiceCatalog#reset() to skip loading cache pools if it's not
a true HDFS file system.
Tests
- Ran tests on S3. Verified that the IllegalStateException doesn't
exists anymore.
Change-Id: Ib243d349177e1b982b313dd6e87ecc2ef4dfc3d8
Reviewed-on: http://gerrit.cloudera.org:8080/16335
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
diff --git a/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java b/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
index f6cfeb0..e51f56f 100644
--- a/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
+++ b/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
@@ -1690,8 +1690,16 @@
// pause the event processing since the cache is anyways being cleared
metastoreEventProcessor_.pause();
// Update the HDFS cache pools
- CachePoolReader reader = new CachePoolReader(true);
- reader.run();
+ try {
+ // We want only 'true' HDFS filesystems to poll the HDFS cache (i.e not S3,
+ // local, etc.)
+ if (FileSystemUtil.getDefaultFileSystem() instanceof DistributedFileSystem) {
+ CachePoolReader reader = new CachePoolReader(true);
+ reader.run();
+ }
+ } catch (IOException e) {
+ LOG.error("Couldn't identify the default FS. Cache Pool reader will be disabled.");
+ }
versionLock_.writeLock().lock();
// In case of an empty new catalog, the version should still change to reflect the
// reset operation itself and to unblock impalads by making the catalog version >
