tree 55988a538700626fb8e8c44e3f77d5a701fad3b3
parent 3193d8c7938741e154320756c444f77992083106
author Steve Loughran <stevel@cloudera.com> 1606314662 +0000
committer GitHub <noreply@github.com> 1606314662 +0000
gpgsig -----BEGIN PGP SIGNATURE-----
 
 wsBcBAABCAAQBQJfvmqmCRBK7hj4Ov3rIwAAdHIIABvhFsT+8YGGF5mByckBIZoI
 T9c+BllLfvQ6ksxsBR2WNsEwwHkHUXZrMORd8PXwrcJnrl8yZO4iqrS97CMvAtbZ
 9/JxiBR/2Qz7h8lTMLwFGPPtHjzF+5VUYR/YV4wFWzMzTIrXYrow45z3gMaRXAb3
 Q3zC4X8Y8igcjh1LnC8qfPhB+0Y5XM0YOmLI4YlnUSSIp8jDvythkp4DFhn1LYKV
 sHGgGGpL8F9CJZKX3VvEnRd9sckb1sxg0tD1YwVKp7bJRQkF6/AALarbxDG0pT2v
 I2f9BOBb+uWHmrsmyGAujPd6Rvf8T+DL/8HFtDChZ5AnoKvL7TfJOKiVZis6ZZo=
 =YBz5
 -----END PGP SIGNATURE-----
 

HADOOP-17313. FileSystem.get to support slow-to-instantiate FS clients. (#2396)


This adds a semaphore to throttle the number of FileSystem instances which
can be created simultaneously, set in "fs.creation.parallel.count".

This is designed to reduce the impact of many threads in an application calling
FileSystem.get() on a filesystem which takes time to instantiate -for example
to an object where HTTPS connections are set up during initialization.
Many threads trying to do this may create spurious delays by conflicting
for access to synchronized blocks, when simply limiting the parallelism
diminishes the conflict, so speeds up all threads trying to access
the store.

The default value, 64, is larger than is likely to deliver any speedup -but
it does mean that there should be no adverse effects from the change.

If a service appears to be blocking on all threads initializing connections to
abfs, s3a or store, try a smaller (possibly significantly smaller) value.

Contributed by Steve Loughran.