tree 694d8c3e403105c1b4613732f280bfe6a7e8db4c
parent 51b85811bc4d7a9b3595bc04283c8cf6c7172d8a
author Enrico Minack <github@enrico.minack.dev> 1710243371 +0100
committer GitHub <noreply@github.com> 1710243371 +0800
gpgsig -----BEGIN PGP SIGNATURE-----
 
 wsFcBAABCAAQBQJl8D4rCRC1aQ7uu5UhlAAAbhgQAHaVkjNdZJgqiQgnJRZOo916
 /Y1eDcu5ctNNC5ZHA0Ou36KjrW/PvaCcczoW77d/OGv+U3rUzcF3srWAwGHl0tYH
 aULcl7+GFLIW5EhCIBNxtf3A4T5MLfrw2oYs/C2sYrxqEHPUjP4fZv2iQj7CctPh
 TLQKGJX7P4oGYoB4MXzCZSJKRMWOoQhAZcSai34N28tU0wXx/XFqK2KyRnTGaGf1
 vhyIJJNxIrDvC/9M6VUCmmGegmK85f8g61GrzYVve+/AZqks+Ip+JOgXL2Grst1Z
 fA3zybvQL860AcqFv9tO496Olns/E4XxaT59x5XrJS99Ief4gmi/4BAztO4rStPX
 a5fJqGXhBvutQTRBA4NdNO9UszlHDooWQG0+6U18TKfD8mQppK5j8FyBhILhPni+
 JwEqNZzrUCRZL2c+wwNzJvBAaWhx/tbLQTkvmQ2NHsQahiWUbm1VvQJKUXxjThC0
 h3ibrccBFQEnfHD42UhOtVRNQ9qh51pY7bfG736wa5FqGt/nF39+jrvCJJaqqc1U
 TH5T7RSYCDNpmxQmqK2lrKL/BN3ytjQIJHY0mEI/H/zv3Yh6ARWGl4/gLJ0FxM/h
 +cNtzA7xScYRPLxCYkOz3uT2F1ixQi7JUueipsq9O8/lDlm4inJjc/IX4FaHWvnb
 njSi5a7iYni6l5U4+BHo
 =oPiC
 -----END PGP SIGNATURE-----
 

[#731][FOLLOWUP] feat(Spark): Configure blockIdLayout for Spark based on max partitions (#1566)

### What changes were proposed in this pull request?
The configuration of the block id layout for Spark2 and Spark3 can be simplified by only providing the maximal number of partitions. Increments `SHUFFLE_SERVER_VERSION` as the "new" Spark client requires to connect to a "new" shuffle server.

### Why are the changes needed?
Currently, optimally configuring the block id layout for Spark is quite complex: https://github.com/apache/incubator-uniffle/pull/1528/files#diff-09ce7eaa98815d62ca00b2a8b0a45b0a922b047014c1f91dc17081b3fef8e7a8R106-R112

Three values have to be provided: the number of bits for sequence number, partition id and task attempt id. The task attempt id has to provide more bits than the partition id, in the sense that the maximal number of attempts can be stored. This requires to also account for speculative execution.

The RssShuffleManager can do the computation and derive the optimal block id layout configuration from the maximal number of partitions only.

### Does this PR introduce _any_ user-facing change?
Adds optional configuration `spark.rss.blockId.maxPartitions`.

### How was this patch tested?
Unit and integration tests.