tree d06710c66b51fdefe6e9210950fd35808425e3a8
parent a5490281ee122cc3b6aa1c50e5cfc0e53bba0f77
author RickyMa <rickyma@tencent.com> 1708763431 +0800
committer GitHub <noreply@github.com> 1708763431 +0800
gpgsig -----BEGIN PGP SIGNATURE-----
 
 wsFcBAABCAAQBQJl2aknCRC1aQ7uu5UhlAAAQRYQAKarbcW9oy2Z2Wd8ztI0rpvA
 FY8Y5ldXyLi0z8OMw3J5SN4C8XGPiWHkld3fe4wBOBLs5UibTDYohobCNxds3rru
 /haXwsDrJqWdJ176LYfY1AndntJd/19klJ9t8+5TSE+AYM/TFE6uvrabtNHCjrbz
 VPK8oA8VFTwVBEmnPUoWQy+QN22B4YXMPaIyy6jdiETJetu5ZFUNK5icSs2NIc7S
 VX6p8beD+NxfYfSvktjv9U2nGITSZW6bL/O1eEx7/IFlWlng+RryHSaCA+vh7Du1
 ZpAFqu1ElLkgDvYuq+DSoDnpp9h/7gxDNIcjtOkqVc9N7xAws8NCz6H8EvSFDVqP
 Jk/dPVusfL2qX2A3BBbVS8HKu5Aay3Q9/Nwg703I7JdzKfm1Jl5TCxLVzGVLbhiv
 R39eFNgXyoBbTKBsCiAc37cVdtQ+4uP4B4DAV68ckXadNKuaiH+WOaLAcsJLG7J3
 k7fBrdC5riOwNQyLwRySbXQtnqOXGwDsWR1IMMyd70f8gs/wAGV2WnIIgG5duaYu
 cohmaQenfn9f6WBYtdMAfzuE5SP9u4/Wq56QfJmbqnUWdrlIpj0w3aumn/mMz4uq
 k9eqZANkVfYW3FS7jWDEdvZYDC6qUtVli5GQjn+CvOzcDWo2HpIDXuv3FQV8uesz
 rTK3uJ7hMceVWfuKnR16
 =2Dn/
 -----END PGP SIGNATURE-----
 

[#1459] fix(server): Memory leak for exceptional scenarios when flushing events (#1537)

### What changes were proposed in this pull request?

Memory may leak in exceptional scenarios when flushing events

### Why are the changes needed?

A follow-up PR for: https://github.com/apache/incubator-uniffle/pull/1459

I found out that memory may leak in exceptional scenarios when flushing events.
<img width="560" alt="企业微信截图_17086201359387" src="https://github.com/apache/incubator-uniffle/assets/13834479/e7b77e82-4336-4123-b59a-621346edd613">
<img width="811" alt="企业微信截图_1708620172540" src="https://github.com/apache/incubator-uniffle/assets/13834479/931eeb38-0c1b-4cfb-83be-485b74c10e17">
Because the following code snippets have not been executed:
```
event.addCleanupCallback(
  () -> {
    this.clearInFlushBuffer(event.getEventId());
    spBlocks.forEach(spb -> spb.getData().release());
  }
);
```
and
```
event.addCleanupCallback(() -> releaseMemory(event.getSize(), true, false));
```


It can also lead to heap memory not being released, as `HybridStorageManager`.`eventOfUnderStorageManagers` will hold a large amount of unreleased heap memory. This is because the code `event.addCleanupCallback(() -> eventOfUnderStorageManagers.invalidate(event))` has not been executed.
<img width="986" alt="企业微信截图_17086622982256" src="https://github.com/apache/incubator-uniffle/assets/13834479/98721112-8805-4f40-83ef-8daa463b6547">
<img width="454" alt="企业微信截图_17086635217020" src="https://github.com/apache/incubator-uniffle/assets/13834479/e54b9e77-33de-44dd-b727-55c41bdfc95f">



After this PR, the memory will not leak and the exception's stack trace will be something like:
[2024-02-23 14:25:02.614] [LocalFileFlushEventThreadPool-54] [ERROR] DefaultFlushEventHandler.handleEventAndUpdateMetrics - Unexpected exceptions happened due to
java.lang.NullPointerException
        at org.apache.uniffle.server.ShuffleTaskInfo.getMaxConcurrencyPerPartitionToWrite(ShuffleTaskInfo.java:109)
        at org.apache.uniffle.server.ShuffleFlushManager.getMaxConcurrencyPerPartitionWrite(ShuffleFlushManager.java:198)
        at org.apache.uniffle.server.ShuffleFlushManager.processFlushEvent(ShuffleFlushManager.java:149)
        at org.apache.uniffle.server.DefaultFlushEventHandler.handleEventAndUpdateMetrics(DefaultFlushEventHandler.java:87)
        at org.apache.uniffle.server.DefaultFlushEventHandler.lambda$dispatchEvent$0(DefaultFlushEventHandler.java:203)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

We can talk about the above exception in another issue after this PR is merged. This PR is focused on fixing potential memory leaks.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Existing UTs.

---------

Co-authored-by: Enrico Minack <github@enrico.minack.dev>