Contributing

This project is a subsidiary project of Apache Hadoop

All governance is by the Apache Software Foundation, including release process.

To discuss the project, please get on the common developer mailing list.

Do read the Hadoop How to Contribute wiki page.

Issues and Patches

  1. Please create issues on the github project, rather than JIRA.
  2. Create an issue before creating a Pull Request, referring to the issue number in the PR.

Testing

All PRs will need a local test run against an S3 service endpoint of your choice. It doesn‘t have to be AWS S3 standard: diversity is always interesting, what is key is that it’s there.

You MUST declare what S3 endpoint you tested against, such as “S3 Frankfurt” or “Local Minio docker image”.

See Testing the S3A filesystem client and its features for instructions.

Place your secrets (or better, an XInclude reference to them elsewhere in your local filesystem) in the file src/test/resources/auth-keys.xml. DO NOT place this file under revision control. If you do ever commit cloud credentials to any revision control system: rotate the keys. That‘s even if that’s your private internal repo: it‘s all to easy to accidentally make that public. Your IT team will be happier with you saying "I just committed my keys, can I replace them even though I haven’t pushed those changes anywhere" to having to deal with the consequences of a leak of keys.

Consider also using a restricted role for the credentials you are storing in XML/JCEKs files. They only need R/W access to test buckets and the ability to create session credentials for ITestS3ASessionKeys.

If testing against third party stores or when running all tests with temporary credentials, session credentials will not be available. Set test.fs.s3a.sts.enabled to false to disable tests in ITestS3ASessionKeys.

Roadmap: Whatever we need to debug and improve client cloud connectivity

  • Exploration of higher-performance IO.
  • Diagnostics/testing of integration with foundational Hadoop operations.
  • Improving CLI testing with various probes designed to be invoked in a shell and fail with meaningful exit codes. E.g: verifying that a filesystem has a specific (key, val) property, that a specific env var made it through.
  • Something to scan hadoop installations for duplicate artifacts, which knowledge of JARS which main contain the same classes (aws-shaded, aws-s3, etc), and the knowledge of required consistencies (hadoop-, jackson-).

Contributions through PRs welcome.

Bug reports: please include environment and ideally patches.