tree 9d2ae6af8312614b428346220f886a7ecac263be
parent bc52f9f0e582474501e92e6a281f0110754a8af1
author Pavel Solodovnikov <psolodovnikov@querifylabs.com> 1660850185 +0300
committer GitHub <noreply@github.com> 1660850185 +0200
gpgsig -----BEGIN PGP SIGNATURE-----
 
 wsBcBAABCAAQBQJi/pAJCRBK7hj4Ov3rIwAA7LMIAD36HIQZfoZ5DRxT+NqFHXJd
 gzU8Dn2t8Nri+fda5D8sN2VIQ20cLDz/CgjAUeGZoiKgb5KPVYTcZSU+ve2+AVMx
 Qynk/8aQcXRXCg+XBDRLG4L8hLp5N18vIu3chKnfQlndWCzsx8vUNr9w4v8+rp+e
 zkma7mI3Ion2CCFaNT/YsQzy3dHxRWI1Ukkn48wH7YKruPwpoovxEitjsfpjBIsW
 3k4tlj9H3nMKsFHCqIe8+CFb8uyrHVPMfB5RChBHvtxD5diDcvuKywcYFT1nry0d
 egSvGd73Jyz4+hk6R8UC2YNPl7sJWUdlqB2hreQ+3JMYw/3azz0rA7h1QzCcvfw=
 =/PmD
 -----END PGP SIGNATURE-----
 

ARROW-17306: [C++] Provide an optimized `GetFileInfoGenerator` specialization for `LocalFileSystem` (#13796)

Introduce a specialization of `GetFileInfoGenerator` in the `LocalFileSystem` class.

This implementation tries to improves performance by hiding latencies at two levels:
1. Child directories can be readahead so that listing directories entries from disk can be achieved in parallel with other work;
2. Directory entries can be `stat`'ed and yielded in chunks so that the `FileInfoGenerator` consumer can start receiving entries before a large directory is fully processed.

Both mechanisms can be tuned using dedicated parameters in `LocalFileSystemOptions`.

Signed-off-by: Pavel Solodovnikov <pavel.al.solodovnikov@gmail.com>
Co-Authored-by: Igor Seliverstov <iseliverstov@querifylabs.com>

Lead-authored-by: Pavel Solodovnikov <pavel.al.solodovnikov@gmail.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>