| ================================================================================================ |
| Coalesced RDD , large scale |
| ================================================================================================ |
| |
| OpenJDK 64-Bit Server VM 21.0.8+9-LTS on Linux 6.11.0-1018-azure |
| AMD EPYC 7763 64-Core Processor |
| Coalesced RDD: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative |
| ---------------------------------------------------------------------------------------------------------------------------- |
| Coalesce Num Partitions: 100 Num Hosts: 1 258 289 44 0.4 2576.1 1.0X |
| Coalesce Num Partitions: 100 Num Hosts: 5 134 135 2 0.7 1337.3 1.9X |
| Coalesce Num Partitions: 100 Num Hosts: 10 115 119 5 0.9 1153.1 2.2X |
| Coalesce Num Partitions: 100 Num Hosts: 20 121 122 2 0.8 1205.8 2.1X |
| Coalesce Num Partitions: 100 Num Hosts: 40 104 111 8 1.0 1042.2 2.5X |
| Coalesce Num Partitions: 100 Num Hosts: 80 102 109 9 1.0 1023.5 2.5X |
| Coalesce Num Partitions: 500 Num Hosts: 1 861 877 21 0.1 8610.7 0.3X |
| Coalesce Num Partitions: 500 Num Hosts: 5 274 276 2 0.4 2743.6 0.9X |
| Coalesce Num Partitions: 500 Num Hosts: 10 184 185 1 0.5 1838.5 1.4X |
| Coalesce Num Partitions: 500 Num Hosts: 20 147 152 6 0.7 1472.2 1.7X |
| Coalesce Num Partitions: 500 Num Hosts: 40 125 126 1 0.8 1249.5 2.1X |
| Coalesce Num Partitions: 500 Num Hosts: 80 125 126 2 0.8 1250.0 2.1X |
| Coalesce Num Partitions: 1000 Num Hosts: 1 1616 1624 10 0.1 16161.1 0.2X |
| Coalesce Num Partitions: 1000 Num Hosts: 5 423 426 5 0.2 4229.1 0.6X |
| Coalesce Num Partitions: 1000 Num Hosts: 10 264 266 2 0.4 2641.7 1.0X |
| Coalesce Num Partitions: 1000 Num Hosts: 20 182 191 15 0.6 1816.1 1.4X |
| Coalesce Num Partitions: 1000 Num Hosts: 40 135 140 5 0.7 1352.6 1.9X |
| Coalesce Num Partitions: 1000 Num Hosts: 80 137 138 1 0.7 1373.0 1.9X |
| Coalesce Num Partitions: 5000 Num Hosts: 1 7384 7391 6 0.0 73841.7 0.0X |
| Coalesce Num Partitions: 5000 Num Hosts: 5 1899 1911 16 0.1 18986.8 0.1X |
| Coalesce Num Partitions: 5000 Num Hosts: 10 1004 1010 6 0.1 10035.9 0.3X |
| Coalesce Num Partitions: 5000 Num Hosts: 20 571 575 3 0.2 5714.0 0.5X |
| Coalesce Num Partitions: 5000 Num Hosts: 40 349 351 3 0.3 3485.6 0.7X |
| Coalesce Num Partitions: 5000 Num Hosts: 80 232 234 2 0.4 2324.0 1.1X |
| Coalesce Num Partitions: 10000 Num Hosts: 1 13891 13911 35 0.0 138910.9 0.0X |
| Coalesce Num Partitions: 10000 Num Hosts: 5 3828 3830 2 0.0 38275.2 0.1X |
| Coalesce Num Partitions: 10000 Num Hosts: 10 1921 1935 13 0.1 19206.1 0.1X |
| Coalesce Num Partitions: 10000 Num Hosts: 20 1018 1025 6 0.1 10184.3 0.3X |
| Coalesce Num Partitions: 10000 Num Hosts: 40 571 577 9 0.2 5710.2 0.5X |
| Coalesce Num Partitions: 10000 Num Hosts: 80 341 345 3 0.3 3413.2 0.8X |
| |
| |
