tree ccf95a98d27e47793e5ade68ca59d2f207c4ac7f
parent 1ff121041c647f0a7980b59e06335faac9dc2f45
author Steve Loughran <stevel@cloudera.com> 1661336299 +0100
committer GitHub <noreply@github.com> 1661336299 +0100
gpgsig -----BEGIN PGP SIGNATURE-----
 
 wsBcBAABCAAQBQJjBfrrCRBK7hj4Ov3rIwAAA1IIAGmLIU5X2OPNeIv7DNZ6gc5a
 sRSkOKmz6z3Woakl7igX+PX9YK3DDtwwj6uz9Da9G+N5dxFFyX2Ijtkf7Q4CwZ2x
 z9Lva7Tkpn3gGO+857A30K+UBeKQppRbXkoaXBvc6DMDaorOWT3WocdTd9F3+Eyy
 g/qjtTnqeTF0beyzveEcYYDQZ3Xkdjr987alMFhNDiAvfs1wtwouqMp01kzCKfU2
 cl4EIqnn1ComaMgmEdIiOx272c30kbRve3I5UTJjdAQdkdIBX3u7ASSp0PBZsSV6
 dthQEp5yOr1zCAc/gY+hra8AuaevTQtujekiNNzqycpv9EkRmKGn4aLLbDRIRHA=
 =o0Jv
 -----END PGP SIGNATURE-----
 

MAPREDUCE-7403. manifest-committer dynamic partitioning support. (#4728)


Declares its compatibility with Spark's dynamic
output partitioning by having the stream capability
"mapreduce.job.committer.dynamic.partitioning"

Requires a Spark release with SPARK-40034, which
does the probing before deciding whether to 
accept/rejecting instantiation with
dynamic partition overwrite set

This feature can be declared as supported by
any other PathOutputCommitter implementations
whose algorithm and destination filesystem
are compatible.

None of the S3A committers are compatible.

The classic FileOutputCommitter is, but it
does not declare itself as such out of our fear
of changing that code. The Spark-side code
will automatically infer compatibility if
the created committer is of that class or
a subclass.

Contributed by Steve Loughran.